Jun 08 11:36:30.624821 ip-10-0-137-95 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jun 08 11:36:30.624838 ip-10-0-137-95 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jun 08 11:36:30.624847 ip-10-0-137-95 systemd[1]: kubelet.service: Failed with result 'resources'. Jun 08 11:36:30.625302 ip-10-0-137-95 systemd[1]: Failed to start Kubernetes Kubelet. Jun 08 11:36:39.641023 ip-10-0-137-95 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jun 08 11:36:39.641040 ip-10-0-137-95 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8ade1de905794193b708b1ee4f003700 -- Jun 08 11:39:00.569044 ip-10-0-137-95 systemd[1]: Starting Kubernetes Kubelet... Jun 08 11:39:01.018858 ip-10-0-137-95 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 08 11:39:01.018858 ip-10-0-137-95 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jun 08 11:39:01.018858 ip-10-0-137-95 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 08 11:39:01.018858 ip-10-0-137-95 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jun 08 11:39:01.018858 ip-10-0-137-95 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 08 11:39:01.019934 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.019792 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jun 08 11:39:01.027752 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027724 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:39:01.027752 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027748 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:39:01.027752 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027752 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:39:01.027752 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027756 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:39:01.027752 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027759 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:39:01.027752 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027762 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027765 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027767 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027770 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027773 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027776 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027778 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027781 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027783 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027785 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027788 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027791 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027794 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027796 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027798 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027801 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027804 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027807 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027810 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027812 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:39:01.027996 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027815 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027817 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027820 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027822 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027825 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027827 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027830 2562 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027832 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027835 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027837 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027839 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027842 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027845 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027847 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027851 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027855 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027857 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027860 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027862 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027865 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:39:01.028471 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027867 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027870 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027873 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027875 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027877 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027880 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027882 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027885 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027887 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027890 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027893 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027895 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027898 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027900 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027902 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027905 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027907 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027910 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027912 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027915 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:39:01.029076 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027918 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027920 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027923 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027925 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027928 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027932 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027936 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027939 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027942 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027945 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027948 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027950 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027953 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027955 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027959 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027961 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027964 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027966 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027970 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:39:01.029556 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027973 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027978 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027981 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027984 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027986 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027989 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027991 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027994 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027996 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.027999 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028002 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028006 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028009 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028011 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028014 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028740 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028747 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028750 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028753 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028756 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:39:01.030029 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028759 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028762 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028765 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028767 2562 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028770 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028773 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028775 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028778 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028780 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028783 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028785 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028788 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028791 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028793 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028798 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028802 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028804 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028807 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028809 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:39:01.030518 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028812 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028815 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028817 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028820 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028823 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028825 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028828 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028830 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028833 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028835 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028839 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028841 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028844 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028847 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028849 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028852 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028854 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028857 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028859 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028862 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:39:01.031015 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028864 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028867 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028869 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028872 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028874 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028877 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028882 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028886 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028889 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028892 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028894 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028898 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028900 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028903 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028906 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028908 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028912 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028914 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028917 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:39:01.031506 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028920 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028922 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028925 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028928 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028930 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028933 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028935 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028938 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028940 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028943 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028945 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028947 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028950 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028952 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028955 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028957 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028960 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028963 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028966 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028969 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:39:01.031987 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028973 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028975 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028978 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028981 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028985 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028988 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028991 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028993 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028996 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.028999 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029001 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029004 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029007 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029010 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029012 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029015 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029407 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029411 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029414 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029417 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:39:01.032469 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029420 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029422 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029425 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029427 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029430 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029433 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029435 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029438 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029440 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029443 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029445 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029448 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029451 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029454 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029457 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029459 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029462 2562 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029464 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029467 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029469 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:39:01.032992 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029472 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029474 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029476 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029479 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029481 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029484 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029487 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029489 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029492 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029494 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029497 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029499 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029502 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029504 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029507 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029509 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029513 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029516 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029519 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:39:01.033477 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029521 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029524 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029526 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029529 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029531 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029534 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029537 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029539 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029542 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029544 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029548 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029550 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029552 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029555 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029557 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029560 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029562 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029565 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029568 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029570 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:39:01.033955 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029573 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029575 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029578 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029580 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029583 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029587 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029589 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029592 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029595 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029598 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029602 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029605 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029623 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029626 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029628 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029631 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029634 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029636 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029639 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:39:01.034493 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029642 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029650 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029652 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029655 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029657 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029660 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029663 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029665 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029668 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029670 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029672 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029675 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029677 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029680 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029682 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029685 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.029689 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029768 2562 flags.go:64] FLAG: --address="0.0.0.0" Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029775 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029781 2562 flags.go:64] FLAG: --anonymous-auth="true" Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029785 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jun 08 11:39:01.034970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029789 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029792 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029796 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029800 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029803 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029806 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029809 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029813 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029816 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029819 2562 flags.go:64] FLAG: --cgroup-root="" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029822 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029825 2562 flags.go:64] FLAG: --client-ca-file="" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029830 2562 flags.go:64] FLAG: --cloud-provider="external" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029833 2562 flags.go:64] FLAG: --cluster-dns="[]" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029838 2562 flags.go:64] FLAG: --cluster-domain="" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029841 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029844 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029847 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029850 2562 flags.go:64] FLAG: --container-log-max-files="5" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029854 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029856 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029859 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029862 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029865 2562 flags.go:64] FLAG: --contention-profiling="false" Jun 08 11:39:01.035501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029868 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029871 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029874 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029877 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029881 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029884 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029886 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029889 2562 flags.go:64] FLAG: --enable-load-reader="false" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029892 2562 flags.go:64] FLAG: --enable-server="true" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029895 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029900 2562 flags.go:64] FLAG: --event-burst="100" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029902 2562 flags.go:64] FLAG: --event-qps="50" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029905 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029908 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029911 2562 flags.go:64] FLAG: --eviction-hard="" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029915 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029918 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029920 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029924 2562 flags.go:64] FLAG: --eviction-soft="" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029926 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029930 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029934 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029937 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029940 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029942 2562 flags.go:64] FLAG: --fail-swap-on="true" Jun 08 11:39:01.036086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029945 2562 flags.go:64] FLAG: --feature-gates="" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029948 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029951 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029954 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029957 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029960 2562 flags.go:64] FLAG: --healthz-port="10248" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029963 2562 flags.go:64] FLAG: --help="false" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029965 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029968 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029971 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029974 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029977 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029980 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029983 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029986 2562 flags.go:64] FLAG: --image-service-endpoint="" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029988 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029991 2562 flags.go:64] FLAG: --kube-api-burst="100" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029994 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.029997 2562 flags.go:64] FLAG: --kube-api-qps="50" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030000 2562 flags.go:64] FLAG: --kube-reserved="" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030002 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030005 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030008 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030010 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jun 08 11:39:01.036728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030013 2562 flags.go:64] FLAG: --lock-file="" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030016 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030019 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030024 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030029 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030033 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030036 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030039 2562 flags.go:64] FLAG: --logging-format="text" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030042 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030045 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030048 2562 flags.go:64] FLAG: --manifest-url="" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030050 2562 flags.go:64] FLAG: --manifest-url-header="" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030055 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030058 2562 flags.go:64] FLAG: --max-open-files="1000000" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030062 2562 flags.go:64] FLAG: --max-pods="110" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030064 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030067 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030070 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030073 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030075 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030078 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030081 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030089 2562 flags.go:64] FLAG: --node-status-max-images="50" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030091 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030094 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jun 08 11:39:01.037356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030098 2562 flags.go:64] FLAG: --pod-cidr="" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030101 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746976bfb3a92b93dacc79c151b98fd4c932eef19c3e2c1516b14eebcac9184d" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030107 2562 flags.go:64] FLAG: --pod-manifest-path="" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030110 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030112 2562 flags.go:64] FLAG: --pods-per-core="0" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030115 2562 flags.go:64] FLAG: --port="10250" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030118 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030121 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0612f45a92e33814b" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030124 2562 flags.go:64] FLAG: --qos-reserved="" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030127 2562 flags.go:64] FLAG: --read-only-port="10255" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030130 2562 flags.go:64] FLAG: --register-node="true" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030133 2562 flags.go:64] FLAG: --register-with-taints="" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030137 2562 flags.go:64] FLAG: --registry-burst="10" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030140 2562 flags.go:64] FLAG: --registry-qps="5" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030143 2562 flags.go:64] FLAG: --reserved-cpus="" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030146 2562 flags.go:64] FLAG: --reserved-memory="" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030150 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030153 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030156 2562 flags.go:64] FLAG: --rotate-certificates="false" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030159 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030161 2562 flags.go:64] FLAG: --runonce="false" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030164 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030167 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030169 2562 flags.go:64] FLAG: --seccomp-default="false" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030172 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jun 08 11:39:01.037987 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030175 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030178 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030181 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030183 2562 flags.go:64] FLAG: --storage-driver-password="root" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030186 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030189 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030191 2562 flags.go:64] FLAG: --storage-driver-user="root" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030196 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030198 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030201 2562 flags.go:64] FLAG: --system-cgroups="" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030204 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030209 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030212 2562 flags.go:64] FLAG: --tls-cert-file="" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030214 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030219 2562 flags.go:64] FLAG: --tls-min-version="" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030221 2562 flags.go:64] FLAG: --tls-private-key-file="" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030224 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030227 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030231 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030234 2562 flags.go:64] FLAG: --v="2" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030238 2562 flags.go:64] FLAG: --version="false" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030242 2562 flags.go:64] FLAG: --vmodule="" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030246 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.030249 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030397 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:39:01.038662 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030402 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030405 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030408 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030411 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030413 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030416 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030418 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030421 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030424 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030426 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030429 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030431 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030433 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030436 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030440 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030443 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030445 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030448 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030450 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:39:01.039277 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030453 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030455 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030457 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030460 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030462 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030465 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030469 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030471 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030474 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030477 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030480 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030482 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030485 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030488 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030490 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030493 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030496 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030498 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030501 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030503 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:39:01.039839 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030506 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030508 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030511 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030513 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030516 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030518 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030520 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030524 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030527 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030530 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030532 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030535 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030537 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030540 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030542 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030544 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030547 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030549 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030553 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:39:01.040323 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030557 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030560 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030563 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030566 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030569 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030571 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030575 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030578 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030581 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030584 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030587 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030590 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030592 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030595 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030597 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030600 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030602 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030604 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030621 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030624 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:39:01.040802 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030630 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030633 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030636 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030639 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030641 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030644 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030646 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030649 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030651 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030653 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030657 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030662 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030664 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030667 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030670 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030672 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030675 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030678 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030680 2562 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:39:01.041269 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.030683 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.031384 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.038654 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.038669 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.038689 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.038694 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038764 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038769 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038773 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038775 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038778 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038781 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038784 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038786 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038789 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:39:01.041742 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038791 2562 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038795 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038799 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038802 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038804 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038808 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038810 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038813 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038815 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038818 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038820 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038822 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038825 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038828 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038830 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038839 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038841 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038849 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038852 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:39:01.042111 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038854 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038857 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038860 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038863 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038866 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038868 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038871 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038873 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038876 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038878 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038882 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038885 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038888 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038891 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038893 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038896 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038898 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038901 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038903 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:39:01.042654 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038906 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038908 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038911 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038913 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038915 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038918 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038920 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038923 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038925 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038928 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038935 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038938 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038940 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038943 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038946 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038949 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038951 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038954 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038956 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038959 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:39:01.043108 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038961 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038963 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038966 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038968 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038971 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038973 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038975 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038978 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038980 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038983 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038985 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038988 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038990 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038992 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038995 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038997 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.038999 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039002 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039004 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:39:01.043576 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039007 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039010 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039014 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039016 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039025 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039028 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039030 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039033 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039036 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039038 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039041 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039043 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039045 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.039050 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039196 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 08 11:39:01.044083 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039204 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039207 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039210 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039212 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039215 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039219 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039223 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039226 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039228 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039231 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039233 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039236 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039239 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039241 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039243 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039246 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039249 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039252 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039254 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 08 11:39:01.044456 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039256 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039259 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039261 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039269 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039272 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039275 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039277 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039280 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039282 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039285 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039287 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039290 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039293 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039295 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039297 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039300 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039302 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039304 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039307 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039309 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 08 11:39:01.045051 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039311 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039314 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039316 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039318 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039321 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039323 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039325 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039328 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039332 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039335 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039338 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039341 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039343 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039346 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039349 2562 feature_gate.go:328] unrecognized feature gate: Example Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039351 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039359 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039363 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039366 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 08 11:39:01.045562 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039368 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039371 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039373 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039376 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039378 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039381 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039383 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039386 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039388 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039390 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039393 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039395 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039397 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039400 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039402 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039404 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039407 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039409 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039411 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039414 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 08 11:39:01.046031 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039416 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039418 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039421 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039423 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039426 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039428 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039431 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039433 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039436 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039438 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039446 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039449 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039451 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039454 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039456 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039459 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039461 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039464 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039466 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 08 11:39:01.046508 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:01.039468 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 08 11:39:01.047037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.039473 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 08 11:39:01.047037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.039584 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jun 08 11:39:01.047037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.042070 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jun 08 11:39:01.047037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.043770 2562 server.go:1019] "Starting client certificate rotation" Jun 08 11:39:01.047037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.043907 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 08 11:39:01.047037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.043967 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 08 11:39:01.047037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.046892 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 08 11:39:01.058870 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.058839 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 08 11:39:01.071699 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.071681 2562 log.go:25] "Validated CRI v1 runtime API" Jun 08 11:39:01.074699 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.074682 2562 log.go:25] "Validated CRI v1 image API" Jun 08 11:39:01.075532 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.075509 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jun 08 11:39:01.079812 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.079789 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 eafca84d-f02a-447c-a037-757160d29826:/dev/nvme0n1p4 f329e64b-9fc6-4fb7-a034-c1cf6e55ed54:/dev/nvme0n1p3] Jun 08 11:39:01.079907 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.079810 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jun 08 11:39:01.084379 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.084359 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 08 11:39:01.085166 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.085039 2562 manager.go:216] Machine: {Timestamp:2026-06-08 11:39:01.083580918 +0000 UTC m=+0.385357606 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109704 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b50b40467deb924f60e53b63eeeb2 SystemUUID:ec2b50b4-0467-deb9-24f6-0e53b63eeeb2 BootID:8ade1de9-0579-4193-b708-b1ee4f003700 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:17:c2:b8:b2:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:17:c2:b8:b2:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:ce:d7:a1:63:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jun 08 11:39:01.085166 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.085155 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jun 08 11:39:01.085289 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.085237 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.118.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260602-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jun 08 11:39:01.088547 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.088518 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jun 08 11:39:01.088707 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.088550 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-95.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jun 08 11:39:01.090118 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.090107 2562 topology_manager.go:138] "Creating topology manager with none policy" Jun 08 11:39:01.090156 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.090122 2562 container_manager_linux.go:309] "Creating device plugin manager" Jun 08 11:39:01.090156 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.090130 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 08 11:39:01.090156 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.090144 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 08 11:39:01.091440 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.091429 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jun 08 11:39:01.095907 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.095895 2562 state_mem.go:36] "Initialized new in-memory state store" Jun 08 11:39:01.096015 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.096005 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jun 08 11:39:01.096736 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.096726 2562 kubelet.go:486] "Attempting to sync node with API server" Jun 08 11:39:01.096773 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.096747 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jun 08 11:39:01.096773 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.096766 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jun 08 11:39:01.096825 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.096779 2562 kubelet.go:398] "Adding apiserver pod source" Jun 08 11:39:01.096825 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.096788 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jun 08 11:39:01.102761 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.102747 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 08 11:39:01.102814 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.102777 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 08 11:39:01.105792 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.105773 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jun 08 11:39:01.106775 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.106732 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jun 08 11:39:01.106882 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.106863 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jun 08 11:39:01.109214 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.109196 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jun 08 11:39:01.109278 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.109230 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jun 08 11:39:01.110783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110769 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110787 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110794 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110799 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110809 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110814 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110823 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110828 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110835 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110841 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jun 08 11:39:01.110853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110858 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jun 08 11:39:01.111139 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.110863 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jun 08 11:39:01.111724 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.111712 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jun 08 11:39:01.111724 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.111722 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jun 08 11:39:01.114860 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.114847 2562 server.go:1268] "Started kubelet" Jun 08 11:39:01.115060 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.114955 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jun 08 11:39:01.115060 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.114937 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jun 08 11:39:01.115176 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.115075 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jun 08 11:39:01.115672 ip-10-0-137-95 systemd[1]: Started Kubernetes Kubelet. Jun 08 11:39:01.117269 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.117241 2562 server.go:310] "Adding debug handlers to kubelet server" Jun 08 11:39:01.119940 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.119924 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jun 08 11:39:01.121449 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.121432 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7hsbr" Jun 08 11:39:01.123644 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.123625 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jun 08 11:39:01.123644 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.123637 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-95.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jun 08 11:39:01.124524 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.123425 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-95.ec2.internal.18b71828c9fc16da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-95.ec2.internal,UID:ip-10-0-137-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-95.ec2.internal,},FirstTimestamp:2026-06-08 11:39:01.114820314 +0000 UTC m=+0.416597002,LastTimestamp:2026-06-08 11:39:01.114820314 +0000 UTC m=+0.416597002,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-95.ec2.internal,}" Jun 08 11:39:01.126043 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126025 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jun 08 11:39:01.126043 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126035 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jun 08 11:39:01.126176 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.126098 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.126744 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126724 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jun 08 11:39:01.126845 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126726 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jun 08 11:39:01.126845 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126765 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jun 08 11:39:01.126845 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126843 2562 factory.go:55] Registering systemd factory Jun 08 11:39:01.126845 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126858 2562 factory.go:223] Registration of the systemd container factory successfully Jun 08 11:39:01.127046 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126872 2562 reconstruct.go:97] "Volume reconstruction finished" Jun 08 11:39:01.127046 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.126880 2562 reconciler.go:29] "Reconciler: start to sync state" Jun 08 11:39:01.127121 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.127047 2562 factory.go:153] Registering CRI-O factory Jun 08 11:39:01.127121 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.127059 2562 factory.go:223] Registration of the crio container factory successfully Jun 08 11:39:01.127121 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.127110 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jun 08 11:39:01.127229 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.127131 2562 factory.go:103] Registering Raw factory Jun 08 11:39:01.127229 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.127141 2562 manager.go:1196] Started watching for new ooms in manager Jun 08 11:39:01.127561 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.127546 2562 manager.go:349] Starting recovery of all containers Jun 08 11:39:01.127890 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.127862 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jun 08 11:39:01.127890 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.127876 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jun 08 11:39:01.129045 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.129011 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7hsbr" Jun 08 11:39:01.140030 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.140006 2562 manager.go:354] Recovery completed Jun 08 11:39:01.142759 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.142737 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jun 08 11:39:01.145808 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.145797 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:39:01.148079 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.148059 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:39:01.148157 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.148091 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:39:01.148157 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.148101 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:39:01.148720 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.148706 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:39:01.148774 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.148709 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jun 08 11:39:01.148812 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.148775 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jun 08 11:39:01.148812 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.148807 2562 state_mem.go:36] "Initialized new in-memory state store" Jun 08 11:39:01.149334 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.149320 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:39:01.149382 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.149344 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:39:01.149382 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.149354 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:39:01.150819 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.150806 2562 policy_none.go:49] "None policy: Start" Jun 08 11:39:01.150858 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.150823 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jun 08 11:39:01.150858 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.150833 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jun 08 11:39:01.152671 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.152661 2562 policy_none.go:47] "Start" Jun 08 11:39:01.188168 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.188152 2562 manager.go:337] "Starting Device Plugin manager" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.188183 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.188197 2562 server.go:87] "Starting device plugin registration server" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.188425 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.188435 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.188545 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.188656 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.188671 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.190908 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.190957 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.195790 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.197090 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.197123 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.197141 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.197200 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jun 08 11:39:01.204669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.199526 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:39:01.289253 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.289182 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:39:01.290168 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.290147 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:39:01.290247 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.290181 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:39:01.290247 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.290191 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:39:01.290247 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.290212 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.297261 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.297244 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal"] Jun 08 11:39:01.297314 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.297308 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:39:01.298028 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.298010 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:39:01.298128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.298042 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:39:01.298128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.298053 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:39:01.298248 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.298233 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.298308 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.298259 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-95.ec2.internal\": node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.300389 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.300376 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:39:01.300498 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.300484 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.300539 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.300526 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:39:01.301047 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.301029 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:39:01.301128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.301033 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:39:01.301128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.301088 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:39:01.301128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.301098 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:39:01.301232 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.301066 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:39:01.301232 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.301148 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:39:01.303210 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.303196 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.303281 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.303222 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 08 11:39:01.303857 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.303840 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientMemory" Jun 08 11:39:01.303947 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.303872 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasNoDiskPressure" Jun 08 11:39:01.303947 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.303888 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeHasSufficientPID" Jun 08 11:39:01.312455 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.312438 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.336280 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.336246 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-95.ec2.internal\" not found" node="ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.340451 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.340437 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-95.ec2.internal\" not found" node="ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.412949 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.412925 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.428222 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.428200 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c15c280ea977f6d362edf439f9b89eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"8c15c280ea977f6d362edf439f9b89eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.428319 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.428229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c15c280ea977f6d362edf439f9b89eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"8c15c280ea977f6d362edf439f9b89eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.428319 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.428246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/757d7e7b88cdd3b52ce5ab35fd14a29f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-95.ec2.internal\" (UID: \"757d7e7b88cdd3b52ce5ab35fd14a29f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.513314 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.513268 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.528684 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.528659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c15c280ea977f6d362edf439f9b89eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"8c15c280ea977f6d362edf439f9b89eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.528814 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.528693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c15c280ea977f6d362edf439f9b89eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"8c15c280ea977f6d362edf439f9b89eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.528814 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.528714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/757d7e7b88cdd3b52ce5ab35fd14a29f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-95.ec2.internal\" (UID: \"757d7e7b88cdd3b52ce5ab35fd14a29f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.528814 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.528741 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/757d7e7b88cdd3b52ce5ab35fd14a29f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-95.ec2.internal\" (UID: \"757d7e7b88cdd3b52ce5ab35fd14a29f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.528814 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.528762 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c15c280ea977f6d362edf439f9b89eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"8c15c280ea977f6d362edf439f9b89eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.528814 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.528771 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c15c280ea977f6d362edf439f9b89eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal\" (UID: \"8c15c280ea977f6d362edf439f9b89eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.614180 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.614092 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.642604 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.642581 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.647435 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:01.647244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" Jun 08 11:39:01.714740 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.714543 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.815133 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.815085 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:01.915704 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:01.915593 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.016127 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:02.016097 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.042595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.042572 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jun 08 11:39:02.043186 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.042718 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 08 11:39:02.117243 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:02.117202 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.135121 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.135082 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-07 11:34:01 +0000 UTC" deadline="2028-02-07 23:42:06.829580796 +0000 UTC" Jun 08 11:39:02.135121 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.135112 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14628h3m4.694473669s" Jun 08 11:39:02.179459 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.179351 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:39:02.217658 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:02.217620 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.230634 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:02.230585 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c15c280ea977f6d362edf439f9b89eb.slice/crio-eb7f530ad98abd48c77f915281741f9261684a902618227003adb1c18a3ec7b8 WatchSource:0}: Error finding container eb7f530ad98abd48c77f915281741f9261684a902618227003adb1c18a3ec7b8: Status 404 returned error can't find the container with id eb7f530ad98abd48c77f915281741f9261684a902618227003adb1c18a3ec7b8 Jun 08 11:39:02.231173 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:02.231154 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757d7e7b88cdd3b52ce5ab35fd14a29f.slice/crio-5a43f18ddb9ea29bb17cf970616a9fbba4f75223e16d505fb0efe5206cf56a69 WatchSource:0}: Error finding container 5a43f18ddb9ea29bb17cf970616a9fbba4f75223e16d505fb0efe5206cf56a69: Status 404 returned error can't find the container with id 5a43f18ddb9ea29bb17cf970616a9fbba4f75223e16d505fb0efe5206cf56a69 Jun 08 11:39:02.235519 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.235504 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:39:02.318646 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:02.318590 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.419208 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:02.419172 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.432579 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.432524 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:39:02.519654 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:02.519621 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.620497 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:02.620454 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-95.ec2.internal\" not found" Jun 08 11:39:02.655821 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.655793 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:39:02.727111 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.727022 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" Jun 08 11:39:02.737531 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.737397 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 08 11:39:02.738895 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.738682 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" Jun 08 11:39:02.746760 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.746723 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 08 11:39:02.846985 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:02.846840 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 08 11:39:03.098236 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.098152 2562 apiserver.go:52] "Watching apiserver" Jun 08 11:39:03.104316 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.104284 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jun 08 11:39:03.106967 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.106933 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-h6rx5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal","openshift-multus/multus-additional-cni-plugins-bbxx9","openshift-multus/multus-s4mjk","kube-system/global-pull-secret-syncer-tk87d","kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal","openshift-multus/network-metrics-daemon-6d2j9","openshift-network-diagnostics/network-check-target-hxt4l","openshift-network-operator/iptables-alerter-rjxmg","openshift-ovn-kubernetes/ovnkube-node-xr47f","kube-system/konnectivity-agent-7c57h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p","openshift-cluster-node-tuning-operator/tuned-kn2mj","openshift-dns/node-resolver-qrh9w"] Jun 08 11:39:03.109674 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.109653 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:03.109772 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.109738 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:03.111529 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.111492 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.113533 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.113515 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jun 08 11:39:03.113643 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.113542 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:39:03.113707 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.113672 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lkwrc\"" Jun 08 11:39:03.113829 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.113780 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.115941 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.115761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.115941 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.115882 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jun 08 11:39:03.116235 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.116217 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxmpq\"" Jun 08 11:39:03.116293 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.116268 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jun 08 11:39:03.117678 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.117214 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jun 08 11:39:03.117678 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.117286 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jun 08 11:39:03.117678 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.117286 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jun 08 11:39:03.117678 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.117534 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jun 08 11:39:03.117942 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.117719 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xbn27\"" Jun 08 11:39:03.117996 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.117954 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.118040 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.118011 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:03.120211 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.120190 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.122242 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.122221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fm7j9\"" Jun 08 11:39:03.122354 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.122279 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jun 08 11:39:03.122533 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.122519 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jun 08 11:39:03.122585 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.122545 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.124531 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.124507 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:39:03.124640 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.124581 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jun 08 11:39:03.124726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.124517 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jun 08 11:39:03.124840 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.124817 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4pwk5\"" Jun 08 11:39:03.125101 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.125080 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.127231 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.127214 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jun 08 11:39:03.127412 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.127393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:03.127504 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.127429 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z9869\"" Jun 08 11:39:03.127504 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.127441 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:03.127746 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.127727 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jun 08 11:39:03.127838 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.127746 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jun 08 11:39:03.127904 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.127889 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jun 08 11:39:03.130098 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.130080 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.132349 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.132328 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jun 08 11:39:03.132426 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.132376 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7w7bw\"" Jun 08 11:39:03.132510 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.132499 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jun 08 11:39:03.132671 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.132659 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jun 08 11:39:03.132718 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.132690 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jun 08 11:39:03.133176 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.133155 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jun 08 11:39:03.133649 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.133627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.135019 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.135001 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.136457 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.136495 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nlbdk\"" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137085 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137136 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b88db40-878a-4a55-a597-935a3b17f4fa-host\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137221 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-kubelet\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-etc-kubernetes\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.137333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.137277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-sys\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.138143 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138120 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysconfig\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.138222 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138162 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-host\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.138222 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-conf-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.138346 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76vt\" (UniqueName: \"kubernetes.io/projected/96ad9462-64fc-482b-9657-4f5594b57e53-kube-api-access-x76vt\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.138435 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-lib-modules\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.138536 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f860735-37dd-4726-874c-6ae43f557528-tmp\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.138536 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138516 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-system-cni-dir\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.138719 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.138719 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138652 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-system-cni-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.138719 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-socket-dir-parent\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.138891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqw6\" (UniqueName: \"kubernetes.io/projected/7283a8b3-5f6e-4dc0-addb-cee7c7299122-kube-api-access-fqqw6\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:03.138891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysctl-conf\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.138891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-var-lib-kubelet\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.138909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-cni-bin\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-multus-certs\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-cni-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knx5d\" (UniqueName: \"kubernetes.io/projected/a28607fa-dba1-4707-87e6-c4d10f40e16b-kube-api-access-knx5d\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysctl-d\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-k8s-cni-cncf-io\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-netns\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c86af520-68b1-4f8e-b2cc-23ee36040b53-iptables-alerter-script\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-cnibin\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139594 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-hostroot\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a28607fa-dba1-4707-87e6-c4d10f40e16b-cni-binary-copy\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhcrh\" (UniqueName: \"kubernetes.io/projected/c86af520-68b1-4f8e-b2cc-23ee36040b53-kube-api-access-jhcrh\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-systemd\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-os-release\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.140200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-cni-multus\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ad9462-64fc-482b-9657-4f5594b57e53-tmp-dir\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139844 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139853 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mt648\"" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139881 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139853 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-kubernetes\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-os-release\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-daemon-config\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-dbus\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139951 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-modprobe-d\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-kubelet-config\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.139998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96ad9462-64fc-482b-9657-4f5594b57e53-hosts-file\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.140016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f860735-37dd-4726-874c-6ae43f557528-etc-tuned\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.140036 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b88db40-878a-4a55-a597-935a3b17f4fa-serviceca\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.140058 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-cnibin\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.140085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2lt\" (UniqueName: \"kubernetes.io/projected/4933408b-be9c-45fc-ac36-622024c3d641-kube-api-access-pv2lt\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.141117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.140107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86af520-68b1-4f8e-b2cc-23ee36040b53-host-slash\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.142204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.140128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-run\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.142204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.142034 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgchh\" (UniqueName: \"kubernetes.io/projected/4f860735-37dd-4726-874c-6ae43f557528-kube-api-access-zgchh\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.142204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.142061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mh2g\" (UniqueName: \"kubernetes.io/projected/3b88db40-878a-4a55-a597-935a3b17f4fa-kube-api-access-7mh2g\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.142204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.142082 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.143166 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.143000 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 08 11:39:03.157732 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.157681 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5mnfs" Jun 08 11:39:03.167282 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.167254 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5mnfs" Jun 08 11:39:03.201293 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.201236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" event={"ID":"757d7e7b88cdd3b52ce5ab35fd14a29f","Type":"ContainerStarted","Data":"5a43f18ddb9ea29bb17cf970616a9fbba4f75223e16d505fb0efe5206cf56a69"} Jun 08 11:39:03.202300 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.202275 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" event={"ID":"8c15c280ea977f6d362edf439f9b89eb","Type":"ContainerStarted","Data":"eb7f530ad98abd48c77f915281741f9261684a902618227003adb1c18a3ec7b8"} Jun 08 11:39:03.227595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.227572 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jun 08 11:39:03.243356 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:03.243489 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-cni-bin\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.243489 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-sys-fs\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.243489 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysctl-d\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.243489 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-k8s-cni-cncf-io\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.243672 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243525 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-k8s-cni-cncf-io\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.243672 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-node-log\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.243672 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fw2t\" (UniqueName: \"kubernetes.io/projected/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-kube-api-access-7fw2t\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.243672 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243627 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysctl-d\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.243834 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-netns\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.243834 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c86af520-68b1-4f8e-b2cc-23ee36040b53-iptables-alerter-script\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.243834 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243764 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-netns\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.243834 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243767 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ae4b988-3539-4b07-85ab-690bdfd35c68-agent-certs\") pod \"konnectivity-agent-7c57h\" (UID: \"2ae4b988-3539-4b07-85ab-690bdfd35c68\") " pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.243834 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovn-node-metrics-cert\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovnkube-script-lib\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243872 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-cnibin\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-kubelet\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-ovn\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243917 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-cnibin\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-hostroot\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-var-lib-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.243999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.244020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-slash\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-run-netns\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a28607fa-dba1-4707-87e6-c4d10f40e16b-cni-binary-copy\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhcrh\" (UniqueName: \"kubernetes.io/projected/c86af520-68b1-4f8e-b2cc-23ee36040b53-kube-api-access-jhcrh\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-cni-netd\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-systemd\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-os-release\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-cni-multus\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.244341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ad9462-64fc-482b-9657-4f5594b57e53-tmp-dir\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-systemd-units\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-kubernetes\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-os-release\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-daemon-config\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-etc-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-dbus\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244525 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c86af520-68b1-4f8e-b2cc-23ee36040b53-iptables-alerter-script\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-modprobe-d\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-kubelet-config\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96ad9462-64fc-482b-9657-4f5594b57e53-hosts-file\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244598 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-systemd\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.244685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244681 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-os-release\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.245112 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244886 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96ad9462-64fc-482b-9657-4f5594b57e53-tmp-dir\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.245112 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.244603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-log-socket\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.245112 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245081 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-hostroot\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.245224 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245170 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.245256 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245240 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-kubernetes\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.245474 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovnkube-config\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.245525 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245494 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-env-overrides\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.245525 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-device-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.245683 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f860735-37dd-4726-874c-6ae43f557528-etc-tuned\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.245683 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b88db40-878a-4a55-a597-935a3b17f4fa-serviceca\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.245683 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-cnibin\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.245683 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245623 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a28607fa-dba1-4707-87e6-c4d10f40e16b-cni-binary-copy\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.245683 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2lt\" (UniqueName: \"kubernetes.io/projected/4933408b-be9c-45fc-ac36-622024c3d641-kube-api-access-pv2lt\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.245683 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86af520-68b1-4f8e-b2cc-23ee36040b53-host-slash\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.245683 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-registration-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.245929 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245689 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-os-release\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.245929 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-run\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.245929 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245815 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-run\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.245929 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.245846 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-cni-multus\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.246282 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246257 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-daemon-config\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.246402 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246388 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-modprobe-d\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.246454 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b88db40-878a-4a55-a597-935a3b17f4fa-serviceca\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.246454 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246439 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-dbus\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.246538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgchh\" (UniqueName: \"kubernetes.io/projected/4f860735-37dd-4726-874c-6ae43f557528-kube-api-access-zgchh\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.246538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246480 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-kubelet-config\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.246538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mh2g\" (UniqueName: \"kubernetes.io/projected/3b88db40-878a-4a55-a597-935a3b17f4fa-kube-api-access-7mh2g\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.246538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246508 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96ad9462-64fc-482b-9657-4f5594b57e53-hosts-file\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.246538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.246721 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:03.246721 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.246721 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b88db40-878a-4a55-a597-935a3b17f4fa-host\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.246721 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.246884 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-kubelet\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.246884 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246783 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86af520-68b1-4f8e-b2cc-23ee36040b53-host-slash\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.246884 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-etc-kubernetes\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.246884 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-systemd\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.247016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246883 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jun 08 11:39:03.247016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-run-ovn-kubernetes\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.247016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-sys\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysconfig\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-host\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.246986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-conf-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76vt\" (UniqueName: \"kubernetes.io/projected/96ad9462-64fc-482b-9657-4f5594b57e53-kube-api-access-x76vt\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-socket-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-lib-modules\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f860735-37dd-4726-874c-6ae43f557528-tmp\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-system-cni-dir\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-system-cni-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247213 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-conf-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247223 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-host\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-socket-dir-parent\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqw6\" (UniqueName: \"kubernetes.io/projected/7283a8b3-5f6e-4dc0-addb-cee7c7299122-kube-api-access-fqqw6\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247262 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b88db40-878a-4a55-a597-935a3b17f4fa-host\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.247312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247285 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysctl-conf\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.247321 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.247392 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret podName:ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab nodeName:}" failed. No retries permitted until 2026-06-08 11:39:03.747371814 +0000 UTC m=+3.049148511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret") pod "global-pull-secret-syncer-tk87d" (UID: "ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247418 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-cnibin\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247458 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-etc-kubernetes\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247469 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247485 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-kubelet\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247511 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-sys\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.247522 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.247557 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:03.747544192 +0000 UTC m=+3.049320866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-var-lib-kubelet\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247139 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysconfig\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-cni-bin\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247601 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-socket-dir-parent\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-multus-certs\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247668 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4b988-3539-4b07-85ab-690bdfd35c68-konnectivity-ca\") pod \"konnectivity-agent-7c57h\" (UID: \"2ae4b988-3539-4b07-85ab-690bdfd35c68\") " pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247701 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-var-lib-kubelet\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.247893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247746 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4933408b-be9c-45fc-ac36-622024c3d641-system-cni-dir\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247751 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-etc-sysctl-conf\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247783 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-run-multus-certs\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247788 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfs8x\" (UniqueName: \"kubernetes.io/projected/741d0109-7f8b-429e-b018-14e2c477977c-kube-api-access-wfs8x\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-cni-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247828 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-host-var-lib-cni-bin\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knx5d\" (UniqueName: \"kubernetes.io/projected/a28607fa-dba1-4707-87e6-c4d10f40e16b-kube-api-access-knx5d\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247877 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f860735-37dd-4726-874c-6ae43f557528-lib-modules\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247894 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-system-cni-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247926 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a28607fa-dba1-4707-87e6-c4d10f40e16b-multus-cni-dir\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.247932 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.248898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.248497 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4933408b-be9c-45fc-ac36-622024c3d641-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.250313 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.250285 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f860735-37dd-4726-874c-6ae43f557528-etc-tuned\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.250405 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.250311 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f860735-37dd-4726-874c-6ae43f557528-tmp\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.252931 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.252912 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:39:03.252931 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.252932 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:39:03.253077 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.252942 2562 projected.go:196] Error preparing data for projected volume kube-api-access-n6hq2 for pod openshift-network-diagnostics/network-check-target-hxt4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:03.253077 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.252988 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2 podName:8c5da3e0-9dfe-4570-9c0f-2e804419ed32 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:03.752973499 +0000 UTC m=+3.054750174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n6hq2" (UniqueName: "kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2") pod "network-check-target-hxt4l" (UID: "8c5da3e0-9dfe-4570-9c0f-2e804419ed32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:03.255755 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.255731 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhcrh\" (UniqueName: \"kubernetes.io/projected/c86af520-68b1-4f8e-b2cc-23ee36040b53-kube-api-access-jhcrh\") pod \"iptables-alerter-rjxmg\" (UID: \"c86af520-68b1-4f8e-b2cc-23ee36040b53\") " pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.256736 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.256712 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgchh\" (UniqueName: \"kubernetes.io/projected/4f860735-37dd-4726-874c-6ae43f557528-kube-api-access-zgchh\") pod \"tuned-kn2mj\" (UID: \"4f860735-37dd-4726-874c-6ae43f557528\") " pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.256821 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.256773 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knx5d\" (UniqueName: \"kubernetes.io/projected/a28607fa-dba1-4707-87e6-c4d10f40e16b-kube-api-access-knx5d\") pod \"multus-s4mjk\" (UID: \"a28607fa-dba1-4707-87e6-c4d10f40e16b\") " pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.257744 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.257722 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76vt\" (UniqueName: \"kubernetes.io/projected/96ad9462-64fc-482b-9657-4f5594b57e53-kube-api-access-x76vt\") pod \"node-resolver-qrh9w\" (UID: \"96ad9462-64fc-482b-9657-4f5594b57e53\") " pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.258394 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.258335 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2lt\" (UniqueName: \"kubernetes.io/projected/4933408b-be9c-45fc-ac36-622024c3d641-kube-api-access-pv2lt\") pod \"multus-additional-cni-plugins-bbxx9\" (UID: \"4933408b-be9c-45fc-ac36-622024c3d641\") " pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.258394 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.258346 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqw6\" (UniqueName: \"kubernetes.io/projected/7283a8b3-5f6e-4dc0-addb-cee7c7299122-kube-api-access-fqqw6\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:03.258899 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.258871 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mh2g\" (UniqueName: \"kubernetes.io/projected/3b88db40-878a-4a55-a597-935a3b17f4fa-kube-api-access-7mh2g\") pod \"node-ca-h6rx5\" (UID: \"3b88db40-878a-4a55-a597-935a3b17f4fa\") " pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.349225 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-systemd\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349225 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-run-ovn-kubernetes\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349225 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-socket-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.349225 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4b988-3539-4b07-85ab-690bdfd35c68-konnectivity-ca\") pod \"konnectivity-agent-7c57h\" (UID: \"2ae4b988-3539-4b07-85ab-690bdfd35c68\") " pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349230 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-systemd\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfs8x\" (UniqueName: \"kubernetes.io/projected/741d0109-7f8b-429e-b018-14e2c477977c-kube-api-access-wfs8x\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349263 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-run-ovn-kubernetes\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349296 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-cni-bin\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-sys-fs\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-node-log\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349362 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-cni-bin\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349368 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-socket-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fw2t\" (UniqueName: \"kubernetes.io/projected/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-kube-api-access-7fw2t\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349416 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-sys-fs\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349432 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-node-log\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ae4b988-3539-4b07-85ab-690bdfd35c68-agent-certs\") pod \"konnectivity-agent-7c57h\" (UID: \"2ae4b988-3539-4b07-85ab-690bdfd35c68\") " pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349466 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovn-node-metrics-cert\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovnkube-script-lib\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.349534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-kubelet\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-ovn\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349567 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-var-lib-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-slash\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-run-netns\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349704 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-kubelet\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-cni-netd\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-systemd-units\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349757 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-cni-netd\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-etc-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349792 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-var-lib-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349824 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4b988-3539-4b07-85ab-690bdfd35c68-konnectivity-ca\") pod \"konnectivity-agent-7c57h\" (UID: \"2ae4b988-3539-4b07-85ab-690bdfd35c68\") " pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-log-socket\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovnkube-config\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349863 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-slash\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349880 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-run-netns\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-env-overrides\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-device-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349911 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349826 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-registration-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349953 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-log-socket\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349968 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-etc-selinux\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349986 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-systemd-units\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.349970 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-etc-openvswitch\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.350028 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-run-ovn\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.350055 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-device-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.350071 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/741d0109-7f8b-429e-b018-14e2c477977c-registration-dir\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.350123 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovnkube-script-lib\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.350344 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-env-overrides\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.350827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.350405 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovnkube-config\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.352530 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.352501 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-ovn-node-metrics-cert\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.352660 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.352605 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ae4b988-3539-4b07-85ab-690bdfd35c68-agent-certs\") pod \"konnectivity-agent-7c57h\" (UID: \"2ae4b988-3539-4b07-85ab-690bdfd35c68\") " pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.356940 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.356919 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fw2t\" (UniqueName: \"kubernetes.io/projected/c442b19f-ef6b-49d4-bf2f-e63f4421aab3-kube-api-access-7fw2t\") pod \"ovnkube-node-xr47f\" (UID: \"c442b19f-ef6b-49d4-bf2f-e63f4421aab3\") " pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.357064 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.357008 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfs8x\" (UniqueName: \"kubernetes.io/projected/741d0109-7f8b-429e-b018-14e2c477977c-kube-api-access-wfs8x\") pod \"aws-ebs-csi-driver-node-cbh6p\" (UID: \"741d0109-7f8b-429e-b018-14e2c477977c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.423904 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.423866 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" Jun 08 11:39:03.432526 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.432494 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" Jun 08 11:39:03.444221 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.444202 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s4mjk" Jun 08 11:39:03.449702 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.449683 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:03.458312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.458287 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qrh9w" Jun 08 11:39:03.465801 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.465784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjxmg" Jun 08 11:39:03.474232 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.474207 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h6rx5" Jun 08 11:39:03.481788 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.481767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:03.486421 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.486402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" Jun 08 11:39:03.752445 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.752354 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:03.752445 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.752401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:03.752658 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.752511 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:03.752658 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.752540 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:03.752658 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.752579 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:04.752558764 +0000 UTC m=+4.054335453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:03.752658 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.752598 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret podName:ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab nodeName:}" failed. No retries permitted until 2026-06-08 11:39:04.752588195 +0000 UTC m=+4.054364875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret") pod "global-pull-secret-syncer-tk87d" (UID: "ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:03.852935 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:03.852907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:03.853097 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.853059 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:39:03.853097 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.853077 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:39:03.853097 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.853087 2562 projected.go:196] Error preparing data for projected volume kube-api-access-n6hq2 for pod openshift-network-diagnostics/network-check-target-hxt4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:03.853210 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:03.853146 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2 podName:8c5da3e0-9dfe-4570-9c0f-2e804419ed32 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:04.85312893 +0000 UTC m=+4.154905605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6hq2" (UniqueName: "kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2") pod "network-check-target-hxt4l" (UID: "8c5da3e0-9dfe-4570-9c0f-2e804419ed32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:04.171881 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:04.171811 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-07 11:34:03 +0000 UTC" deadline="2027-11-25 03:12:23.521526625 +0000 UTC" Jun 08 11:39:04.171881 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:04.171843 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12831h33m19.349686214s" Jun 08 11:39:04.198268 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:04.198236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:04.198409 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.198347 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:04.225284 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:04.225190 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741d0109_7f8b_429e_b018_14e2c477977c.slice/crio-bd7fda2dec3e3a5b84a56263a8e834261205ebbfde40e88f8df3be04fcf8ae86 WatchSource:0}: Error finding container bd7fda2dec3e3a5b84a56263a8e834261205ebbfde40e88f8df3be04fcf8ae86: Status 404 returned error can't find the container with id bd7fda2dec3e3a5b84a56263a8e834261205ebbfde40e88f8df3be04fcf8ae86 Jun 08 11:39:04.227222 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:04.227196 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f860735_37dd_4726_874c_6ae43f557528.slice/crio-a32e658e3f862be33dd74b9410449dc1fcb208dfe0787579406ce72e69be3eb4 WatchSource:0}: Error finding container a32e658e3f862be33dd74b9410449dc1fcb208dfe0787579406ce72e69be3eb4: Status 404 returned error can't find the container with id a32e658e3f862be33dd74b9410449dc1fcb208dfe0787579406ce72e69be3eb4 Jun 08 11:39:04.229566 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:04.229540 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae4b988_3539_4b07_85ab_690bdfd35c68.slice/crio-c45fd10f17481059764a0778f193600564c2dad72a438ed007f4b08af670d824 WatchSource:0}: Error finding container c45fd10f17481059764a0778f193600564c2dad72a438ed007f4b08af670d824: Status 404 returned error can't find the container with id c45fd10f17481059764a0778f193600564c2dad72a438ed007f4b08af670d824 Jun 08 11:39:04.230305 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:04.230282 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b88db40_878a_4a55_a597_935a3b17f4fa.slice/crio-faaa725fefec2d616893a0d2517c5a619d2cca9ae4ca5868af1ce8f5cf18f693 WatchSource:0}: Error finding container faaa725fefec2d616893a0d2517c5a619d2cca9ae4ca5868af1ce8f5cf18f693: Status 404 returned error can't find the container with id faaa725fefec2d616893a0d2517c5a619d2cca9ae4ca5868af1ce8f5cf18f693 Jun 08 11:39:04.231112 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:04.231087 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc442b19f_ef6b_49d4_bf2f_e63f4421aab3.slice/crio-61ce4f3b03bfe16b256516fda5a72d26b7c05d41a23dcb40c81120db1067023d WatchSource:0}: Error finding container 61ce4f3b03bfe16b256516fda5a72d26b7c05d41a23dcb40c81120db1067023d: Status 404 returned error can't find the container with id 61ce4f3b03bfe16b256516fda5a72d26b7c05d41a23dcb40c81120db1067023d Jun 08 11:39:04.232863 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:04.232828 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86af520_68b1_4f8e_b2cc_23ee36040b53.slice/crio-0a878174ace84b726fdc8dc5e1789f263660646d1c6faf9df27cf8505bf5a11f WatchSource:0}: Error finding container 0a878174ace84b726fdc8dc5e1789f263660646d1c6faf9df27cf8505bf5a11f: Status 404 returned error can't find the container with id 0a878174ace84b726fdc8dc5e1789f263660646d1c6faf9df27cf8505bf5a11f Jun 08 11:39:04.233784 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:04.233761 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4933408b_be9c_45fc_ac36_622024c3d641.slice/crio-45184fbdf3858f2f1ed64032335f6fa162d0f71d64b068e319e85f309d49e22c WatchSource:0}: Error finding container 45184fbdf3858f2f1ed64032335f6fa162d0f71d64b068e319e85f309d49e22c: Status 404 returned error can't find the container with id 45184fbdf3858f2f1ed64032335f6fa162d0f71d64b068e319e85f309d49e22c Jun 08 11:39:04.762198 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:04.762104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:04.762198 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:04.762166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:04.762391 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.762333 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:04.762448 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.762398 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret podName:ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab nodeName:}" failed. No retries permitted until 2026-06-08 11:39:06.762380129 +0000 UTC m=+6.064156806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret") pod "global-pull-secret-syncer-tk87d" (UID: "ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:04.762912 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.762812 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:04.762912 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.762865 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:06.762850392 +0000 UTC m=+6.064627067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:04.863494 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:04.862914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:04.863494 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.863087 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:39:04.863494 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.863105 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:39:04.863494 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.863118 2562 projected.go:196] Error preparing data for projected volume kube-api-access-n6hq2 for pod openshift-network-diagnostics/network-check-target-hxt4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:04.863494 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:04.863172 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2 podName:8c5da3e0-9dfe-4570-9c0f-2e804419ed32 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:06.863154873 +0000 UTC m=+6.164931563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6hq2" (UniqueName: "kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2") pod "network-check-target-hxt4l" (UID: "8c5da3e0-9dfe-4570-9c0f-2e804419ed32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:05.172658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.172527 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-07 11:34:03 +0000 UTC" deadline="2028-02-07 18:03:43.682221127 +0000 UTC" Jun 08 11:39:05.172658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.172594 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14622h24m38.50963153s" Jun 08 11:39:05.200325 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.200295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:05.200469 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:05.200420 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:05.200834 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.200808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:05.200940 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:05.200917 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:05.217633 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.217428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7c57h" event={"ID":"2ae4b988-3539-4b07-85ab-690bdfd35c68","Type":"ContainerStarted","Data":"c45fd10f17481059764a0778f193600564c2dad72a438ed007f4b08af670d824"} Jun 08 11:39:05.224633 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.224085 2562 generic.go:356] "Generic (PLEG): container finished" podID="8c15c280ea977f6d362edf439f9b89eb" containerID="32ef03d8823cdb746bfd3901e2c2fab21efb98d8ddd1180e1977ad664660731d" exitCode=0 Jun 08 11:39:05.224633 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.224156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" event={"ID":"8c15c280ea977f6d362edf439f9b89eb","Type":"ContainerDied","Data":"32ef03d8823cdb746bfd3901e2c2fab21efb98d8ddd1180e1977ad664660731d"} Jun 08 11:39:05.236550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.236523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerStarted","Data":"45184fbdf3858f2f1ed64032335f6fa162d0f71d64b068e319e85f309d49e22c"} Jun 08 11:39:05.246422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.246021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjxmg" event={"ID":"c86af520-68b1-4f8e-b2cc-23ee36040b53","Type":"ContainerStarted","Data":"0a878174ace84b726fdc8dc5e1789f263660646d1c6faf9df27cf8505bf5a11f"} Jun 08 11:39:05.257188 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.257158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" event={"ID":"4f860735-37dd-4726-874c-6ae43f557528","Type":"ContainerStarted","Data":"a32e658e3f862be33dd74b9410449dc1fcb208dfe0787579406ce72e69be3eb4"} Jun 08 11:39:05.266983 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.266892 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" event={"ID":"741d0109-7f8b-429e-b018-14e2c477977c","Type":"ContainerStarted","Data":"bd7fda2dec3e3a5b84a56263a8e834261205ebbfde40e88f8df3be04fcf8ae86"} Jun 08 11:39:05.269904 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.269873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" event={"ID":"757d7e7b88cdd3b52ce5ab35fd14a29f","Type":"ContainerStarted","Data":"50a85569f96bbaf93be5da23968c248d984b3f3778566b314816c9a47a417abf"} Jun 08 11:39:05.277798 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.277736 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s4mjk" event={"ID":"a28607fa-dba1-4707-87e6-c4d10f40e16b","Type":"ContainerStarted","Data":"aa40984c4cdb0f95dde3db49dc8c09f3821ffcedbf7d074f433199f844a4bc1d"} Jun 08 11:39:05.285520 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.285491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qrh9w" event={"ID":"96ad9462-64fc-482b-9657-4f5594b57e53","Type":"ContainerStarted","Data":"f349dd21fd52273e88346bfe273839edb09853e3a1bbe425a771dc493895d8fe"} Jun 08 11:39:05.304324 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.304297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"61ce4f3b03bfe16b256516fda5a72d26b7c05d41a23dcb40c81120db1067023d"} Jun 08 11:39:05.306505 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:05.306451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h6rx5" event={"ID":"3b88db40-878a-4a55-a597-935a3b17f4fa","Type":"ContainerStarted","Data":"faaa725fefec2d616893a0d2517c5a619d2cca9ae4ca5868af1ce8f5cf18f693"} Jun 08 11:39:06.198200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:06.197601 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:06.198200 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.197743 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:06.345122 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:06.345083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" event={"ID":"8c15c280ea977f6d362edf439f9b89eb","Type":"ContainerStarted","Data":"fd82bacabccbf19398612ab7f54805bca9c6ac4c818af8b50397ef079cabc71c"} Jun 08 11:39:06.360277 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:06.359869 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-95.ec2.internal" podStartSLOduration=4.359848218 podStartE2EDuration="4.359848218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:05.28397648 +0000 UTC m=+4.585753178" watchObservedRunningTime="2026-06-08 11:39:06.359848218 +0000 UTC m=+5.661624916" Jun 08 11:39:06.776397 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:06.776358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:06.776587 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:06.776411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:06.776587 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.776581 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:06.776721 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.776657 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret podName:ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab nodeName:}" failed. No retries permitted until 2026-06-08 11:39:10.776639572 +0000 UTC m=+10.078416251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret") pod "global-pull-secret-syncer-tk87d" (UID: "ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:06.777066 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.777044 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:06.777151 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.777099 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:10.777083082 +0000 UTC m=+10.078859760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:06.878321 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:06.877723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:06.878321 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.877885 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:39:06.878321 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.877907 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:39:06.878321 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.877920 2562 projected.go:196] Error preparing data for projected volume kube-api-access-n6hq2 for pod openshift-network-diagnostics/network-check-target-hxt4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:06.878321 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:06.877976 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2 podName:8c5da3e0-9dfe-4570-9c0f-2e804419ed32 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:10.877958418 +0000 UTC m=+10.179735107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6hq2" (UniqueName: "kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2") pod "network-check-target-hxt4l" (UID: "8c5da3e0-9dfe-4570-9c0f-2e804419ed32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:07.199866 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:07.199747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:07.200304 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:07.199874 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:07.200304 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:07.200227 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:07.200409 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:07.200324 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:08.197660 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:08.197621 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:08.197898 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:08.197756 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:09.197630 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:09.197588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:09.198072 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:09.197728 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:09.198169 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:09.198147 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:09.198294 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:09.198249 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:10.198031 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:10.197997 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:10.198473 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.198127 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:10.813443 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:10.813374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:10.813719 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:10.813449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:10.813719 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.813590 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:10.813719 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.813667 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:18.813652433 +0000 UTC m=+18.115429108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:10.813719 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.813715 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:10.813949 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.813754 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret podName:ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab nodeName:}" failed. No retries permitted until 2026-06-08 11:39:18.813743097 +0000 UTC m=+18.115519773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret") pod "global-pull-secret-syncer-tk87d" (UID: "ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:10.914859 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:10.914828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:10.915006 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.914991 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:39:10.915058 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.915015 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:39:10.915058 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.915027 2562 projected.go:196] Error preparing data for projected volume kube-api-access-n6hq2 for pod openshift-network-diagnostics/network-check-target-hxt4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:10.915118 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:10.915082 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2 podName:8c5da3e0-9dfe-4570-9c0f-2e804419ed32 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:18.915064817 +0000 UTC m=+18.216841492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6hq2" (UniqueName: "kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2") pod "network-check-target-hxt4l" (UID: "8c5da3e0-9dfe-4570-9c0f-2e804419ed32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:11.203298 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:11.203226 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:11.204177 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:11.204044 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:11.204944 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:11.204578 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:11.205183 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:11.205139 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:12.197492 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:12.197437 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:12.197690 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:12.197575 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:13.198441 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:13.198410 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:13.199016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:13.198463 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:13.199016 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:13.198596 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:13.199016 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:13.198752 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:14.198064 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:14.198031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:14.198264 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:14.198162 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:15.197908 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:15.197836 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:15.198303 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:15.197836 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:15.198303 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:15.197945 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:15.198303 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:15.198051 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:16.197975 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:16.197943 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:16.198390 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:16.198043 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:17.197563 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:17.197524 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:17.197749 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:17.197669 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:17.197749 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:17.197735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:17.197871 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:17.197851 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:18.198343 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:18.198298 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:18.198866 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.198451 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:18.875216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:18.875180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:18.875386 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:18.875230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:18.875386 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.875382 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:18.875459 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.875431 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret podName:ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab nodeName:}" failed. No retries permitted until 2026-06-08 11:39:34.875415 +0000 UTC m=+34.177191676 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret") pod "global-pull-secret-syncer-tk87d" (UID: "ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:18.875496 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.875480 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:18.875528 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.875505 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:34.875497277 +0000 UTC m=+34.177273952 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:18.976623 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:18.976519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:18.976782 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.976698 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:39:18.976782 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.976722 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:39:18.976782 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.976735 2562 projected.go:196] Error preparing data for projected volume kube-api-access-n6hq2 for pod openshift-network-diagnostics/network-check-target-hxt4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:18.976934 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:18.976797 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2 podName:8c5da3e0-9dfe-4570-9c0f-2e804419ed32 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:34.976778837 +0000 UTC m=+34.278555523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6hq2" (UniqueName: "kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2") pod "network-check-target-hxt4l" (UID: "8c5da3e0-9dfe-4570-9c0f-2e804419ed32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:19.198072 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:19.198039 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:19.198275 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:19.198043 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:19.198275 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:19.198191 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:19.198275 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:19.198254 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:20.198164 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:20.198127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:20.198438 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:20.198243 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:21.198525 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:21.198489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:21.199011 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:21.198641 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:21.199011 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:21.198676 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:21.199011 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:21.198803 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:22.197787 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:22.197749 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:22.197975 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:22.197880 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:23.198061 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.198028 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:23.198510 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.198038 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:23.198510 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:23.198236 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:23.198510 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:23.198236 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:23.387238 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.387133 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qrh9w" event={"ID":"96ad9462-64fc-482b-9657-4f5594b57e53","Type":"ContainerStarted","Data":"274a5797120e6ced9862af7911d596b08643896b1cf06fd34dafad0fa7dbdd19"} Jun 08 11:39:23.390128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.390028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7c57h" event={"ID":"2ae4b988-3539-4b07-85ab-690bdfd35c68","Type":"ContainerStarted","Data":"ee57424e4bdea020f6705e92c26b9504700fc52f5d35943c9e5c977d9978de99"} Jun 08 11:39:23.392438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.392392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" event={"ID":"4f860735-37dd-4726-874c-6ae43f557528","Type":"ContainerStarted","Data":"29ef41fb17f12fb5d9d5d08cb0a689a26c69f447f26cdfcc8205ee9ad040a82d"} Jun 08 11:39:23.414696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.413996 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qrh9w" podStartSLOduration=3.498243619 podStartE2EDuration="22.413976558s" totalImagesPullingTime="18.915732939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:23.413903958 +0000 UTC m=+22.715680656" watchObservedRunningTime="2026-06-08 11:39:23.413976558 +0000 UTC m=+22.715753253" Jun 08 11:39:23.414696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.414187 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-95.ec2.internal" podStartSLOduration=21.414178844 podStartE2EDuration="21.414178844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:06.360237856 +0000 UTC m=+5.662014555" watchObservedRunningTime="2026-06-08 11:39:23.414178844 +0000 UTC m=+22.715955542" Jun 08 11:39:23.433255 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.433210 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kn2mj" podStartSLOduration=3.508740338 podStartE2EDuration="22.433195442s" totalImagesPullingTime="18.924455104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:23.433197303 +0000 UTC m=+22.734974000" watchObservedRunningTime="2026-06-08 11:39:23.433195442 +0000 UTC m=+22.734972139" Jun 08 11:39:23.450191 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.450141 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7c57h" podStartSLOduration=11.907761624 podStartE2EDuration="22.450124712s" totalImagesPullingTime="10.542363088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:23.449015988 +0000 UTC m=+22.750792685" watchObservedRunningTime="2026-06-08 11:39:23.450124712 +0000 UTC m=+22.751901425" Jun 08 11:39:23.539890 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.539858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:23.620473 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:23.620434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:24.198286 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.198196 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:24.198957 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:24.198342 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:24.298326 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.298281 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:24.375942 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.375902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:24.395913 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.395878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" event={"ID":"741d0109-7f8b-429e-b018-14e2c477977c","Type":"ContainerStarted","Data":"f4c07d80ee8f88094ded1616e6571c6c298fd2cb0e56136c614cc87d15eec261"} Jun 08 11:39:24.397220 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.397199 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s4mjk" event={"ID":"a28607fa-dba1-4707-87e6-c4d10f40e16b","Type":"ContainerStarted","Data":"e52edaf639a0b6ba8a32371c7bad43135430f80c5219885fc98fd9e11b5768d6"} Jun 08 11:39:24.399810 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.399792 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:39:24.400101 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.400081 2562 generic.go:356] "Generic (PLEG): container finished" podID="c442b19f-ef6b-49d4-bf2f-e63f4421aab3" containerID="5ed6155851de86de5e7f3bf043acafb427f74635c839919bc25284a0f96248e4" exitCode=1 Jun 08 11:39:24.400174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.400149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"b476d23e9f1ded8cb9070b87bdd3279e023be6c21f6167f8b192b4775799cc8d"} Jun 08 11:39:24.400217 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.400204 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"7fe5bfec61d14bce159b1d70a8f0da6cab9575bae6090f940d0cf35112f1d3a1"} Jun 08 11:39:24.400259 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.400242 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"6cb23a93095dd30a7c5626b9bffb26823f957aa94d0e2b44ede120026f3e1d52"} Jun 08 11:39:24.400299 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.400280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"b2425d77ded99661dc257c82764d35b571107419aa4bd016f269aa73d0679acd"} Jun 08 11:39:24.400347 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.400309 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerDied","Data":"5ed6155851de86de5e7f3bf043acafb427f74635c839919bc25284a0f96248e4"} Jun 08 11:39:24.400398 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.400348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"9765087a09184d8f977d1b87d5a8670b5a20ae4322128cbc79ed9c61b5544e69"} Jun 08 11:39:24.401467 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.401444 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h6rx5" event={"ID":"3b88db40-878a-4a55-a597-935a3b17f4fa","Type":"ContainerStarted","Data":"c29e0eecaf2ac1f2e0c656971c5b0e146d36ed2041f1da02c2dd4ab4af8e5ff2"} Jun 08 11:39:24.402874 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.402854 2562 generic.go:356] "Generic (PLEG): container finished" podID="4933408b-be9c-45fc-ac36-622024c3d641" containerID="b6c99bf92826c00943f39fadcc60b96f45a2663fc5a18876e6401dfc2ab92638" exitCode=0 Jun 08 11:39:24.402969 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.402944 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerDied","Data":"b6c99bf92826c00943f39fadcc60b96f45a2663fc5a18876e6401dfc2ab92638"} Jun 08 11:39:24.435037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.434987 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s4mjk" podStartSLOduration=4.330894133 podStartE2EDuration="23.434971662s" totalImagesPullingTime="19.104077529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:24.418183901 +0000 UTC m=+23.719960610" watchObservedRunningTime="2026-06-08 11:39:24.434971662 +0000 UTC m=+23.736748342" Jun 08 11:39:24.435515 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.435495 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h6rx5" podStartSLOduration=4.513174885 podStartE2EDuration="23.435487743s" totalImagesPullingTime="18.922312858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:24.435234376 +0000 UTC m=+23.737011072" watchObservedRunningTime="2026-06-08 11:39:24.435487743 +0000 UTC m=+23.737264707" Jun 08 11:39:24.520571 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.520539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:24.622491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.622454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:24.719231 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.719192 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:24.831354 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:24.831325 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jun 08 11:39:25.171302 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.171220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:25.198326 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.198285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:25.198972 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:25.198434 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:25.198972 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.198488 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:25.198972 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:25.198625 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:25.199340 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.199244 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-08T11:39:24.831344981Z","UUID":"c791353d-37b7-4f1a-9490-3f9e336f2a48","Handler":null,"Name":"","Endpoint":""} Jun 08 11:39:25.200998 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.200980 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jun 08 11:39:25.201096 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.201013 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jun 08 11:39:25.256174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.256126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:25.408200 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.408152 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjxmg" event={"ID":"c86af520-68b1-4f8e-b2cc-23ee36040b53","Type":"ContainerStarted","Data":"3c0b061c8f56481993f9e1fb2779664e848fcbb90a1fa1b9e3c68bada31b9d3c"} Jun 08 11:39:25.411100 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.411070 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" event={"ID":"741d0109-7f8b-429e-b018-14e2c477977c","Type":"ContainerStarted","Data":"b834e07085eacada6274688b1c7695bf61ba393a21734dbb16b6f1f78939e869"} Jun 08 11:39:25.425074 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.424986 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rjxmg" podStartSLOduration=5.508855805 podStartE2EDuration="24.424970411s" totalImagesPullingTime="18.916114606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:25.424644587 +0000 UTC m=+24.726421286" watchObservedRunningTime="2026-06-08 11:39:25.424970411 +0000 UTC m=+24.726747109" Jun 08 11:39:25.657042 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.657017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:25.752899 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:25.752836 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:26.197853 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.197817 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:26.198151 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:26.197946 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:26.417126 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.417048 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:39:26.417653 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.417472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"b2fd49ec7b41442669b86956f25b172a2dbb074eff4f0750aebd64704209d668"} Jun 08 11:39:26.419595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.419562 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" event={"ID":"741d0109-7f8b-429e-b018-14e2c477977c","Type":"ContainerStarted","Data":"3b658a83f4ee2c09e6e05d16c93baedc82d72a510265d7b89fe8adb0fa1c947e"} Jun 08 11:39:26.439503 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.439451 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cbh6p" podStartSLOduration=4.488361416 podStartE2EDuration="25.439430732s" totalImagesPullingTime="20.951069316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:26.439130691 +0000 UTC m=+25.740907399" watchObservedRunningTime="2026-06-08 11:39:26.439430732 +0000 UTC m=+25.741207430" Jun 08 11:39:26.453422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.453311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:26.848182 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.848069 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:26.851981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.851956 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:26.852702 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.852677 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:26.908001 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:26.907967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:27.201549 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:27.201472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:27.201731 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:27.201593 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:27.201731 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:27.201653 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:27.201850 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:27.201738 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:27.657056 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:27.656940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:27.755584 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:27.755541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:28.051850 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:28.051769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:28.126800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:28.126761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:28.197675 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:28.197639 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:28.197825 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:28.197773 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:29.198395 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.198216 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:29.198783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.198230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:29.198783 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:29.198466 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:29.198783 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:29.198540 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:29.427164 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.427131 2562 generic.go:356] "Generic (PLEG): container finished" podID="4933408b-be9c-45fc-ac36-622024c3d641" containerID="b7b06606fc059953c6c874674e65a04e76603b9033674d4491bc9530059650dd" exitCode=0 Jun 08 11:39:29.427304 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.427211 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerDied","Data":"b7b06606fc059953c6c874674e65a04e76603b9033674d4491bc9530059650dd"} Jun 08 11:39:29.430744 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.430727 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:39:29.431117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.431094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"7fc2a982f29fa23fb670ba5717569e6776a6f2e939082c0af8ae4e74bcd37a9e"} Jun 08 11:39:29.431322 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.431307 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:29.431388 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.431373 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:29.431537 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.431523 2562 scope.go:117] "RemoveContainer" containerID="5ed6155851de86de5e7f3bf043acafb427f74635c839919bc25284a0f96248e4" Jun 08 11:39:29.447883 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.447868 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:29.997278 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.997054 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:29.997461 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.997445 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jun 08 11:39:29.997552 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:29.997534 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7c57h" Jun 08 11:39:30.101650 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.101604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:30.189503 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.189463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:30.198334 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.198307 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:30.198448 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:30.198421 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:30.434764 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.434729 2562 generic.go:356] "Generic (PLEG): container finished" podID="4933408b-be9c-45fc-ac36-622024c3d641" containerID="fbd2adcebe9bf8876317332fcb6cb46fe35a70b31b05959b4149496341ac8643" exitCode=0 Jun 08 11:39:30.434903 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.434814 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerDied","Data":"fbd2adcebe9bf8876317332fcb6cb46fe35a70b31b05959b4149496341ac8643"} Jun 08 11:39:30.438463 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.438442 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:39:30.438797 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.438775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" event={"ID":"c442b19f-ef6b-49d4-bf2f-e63f4421aab3","Type":"ContainerStarted","Data":"2023f0b75d56b329bb1ae9ffa82bde033d0f6c39d88ce590432df5faf166804e"} Jun 08 11:39:30.439076 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.439045 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:30.454451 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.454432 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:39:30.476164 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.476142 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tk87d"] Jun 08 11:39:30.476249 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.476235 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:30.476353 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:30.476332 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:30.483160 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.483134 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxt4l"] Jun 08 11:39:30.483254 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.483201 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:30.483293 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:30.483268 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:30.484735 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.484718 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6d2j9"] Jun 08 11:39:30.484849 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.484796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:30.484899 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:30.484869 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:30.563495 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.563413 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" podStartSLOduration=10.562276858 podStartE2EDuration="29.563399425s" totalImagesPullingTime="19.001122567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:30.563095363 +0000 UTC m=+29.864872060" watchObservedRunningTime="2026-06-08 11:39:30.563399425 +0000 UTC m=+29.865176193" Jun 08 11:39:30.595833 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:30.595808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:31.442963 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:31.442884 2562 generic.go:356] "Generic (PLEG): container finished" podID="4933408b-be9c-45fc-ac36-622024c3d641" containerID="32470d2970613585d2798e50859aaf5e74d3816572692f0b891fb951893afc76" exitCode=0 Jun 08 11:39:31.443382 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:31.442973 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerDied","Data":"32470d2970613585d2798e50859aaf5e74d3816572692f0b891fb951893afc76"} Jun 08 11:39:32.198196 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:32.198156 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:32.198455 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:32.198250 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:32.198455 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:32.198257 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:32.198455 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:32.198278 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:32.198455 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:32.198368 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:32.198455 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:32.198435 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:34.197951 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:34.197750 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:34.198452 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:34.197780 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:34.198452 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.198016 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxt4l" podUID="8c5da3e0-9dfe-4570-9c0f-2e804419ed32" Jun 08 11:39:34.198452 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:34.197820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:34.198452 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.198102 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:39:34.198452 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.198174 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tk87d" podUID="ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab" Jun 08 11:39:34.896638 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:34.896593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:34.896638 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:34.896640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:34.896872 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.896728 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:34.896872 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.896742 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:34.896872 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.896784 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret podName:ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab nodeName:}" failed. No retries permitted until 2026-06-08 11:40:06.896766485 +0000 UTC m=+66.198543161 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret") pod "global-pull-secret-syncer-tk87d" (UID: "ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab") : object "kube-system"/"original-pull-secret" not registered Jun 08 11:39:34.896872 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.896808 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:40:06.896801024 +0000 UTC m=+66.198577698 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 08 11:39:34.997086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:34.997057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:34.997245 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.997207 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 08 11:39:34.997245 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.997223 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 08 11:39:34.997245 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.997232 2562 projected.go:196] Error preparing data for projected volume kube-api-access-n6hq2 for pod openshift-network-diagnostics/network-check-target-hxt4l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:34.997342 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:34.997279 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2 podName:8c5da3e0-9dfe-4570-9c0f-2e804419ed32 nodeName:}" failed. No retries permitted until 2026-06-08 11:40:06.997266544 +0000 UTC m=+66.299043220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6hq2" (UniqueName: "kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2") pod "network-check-target-hxt4l" (UID: "8c5da3e0-9dfe-4570-9c0f-2e804419ed32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 08 11:39:35.997555 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:35.997520 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-95.ec2.internal" event="NodeReady" Jun 08 11:39:35.998030 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:35.997691 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jun 08 11:39:36.058325 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.058291 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-74ptp"] Jun 08 11:39:36.077705 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.076828 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r7kbj"] Jun 08 11:39:36.077705 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.076989 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.080057 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.080036 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9sblq\"" Jun 08 11:39:36.080187 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.080082 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jun 08 11:39:36.080623 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.080587 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jun 08 11:39:36.102899 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.102876 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7kbj"] Jun 08 11:39:36.102899 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.102903 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-74ptp"] Jun 08 11:39:36.103045 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.102978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.106315 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.106285 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jun 08 11:39:36.106469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.106450 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p49dk\"" Jun 08 11:39:36.106566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.106550 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jun 08 11:39:36.106648 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.106573 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jun 08 11:39:36.197359 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.197329 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:39:36.197504 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.197440 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:39:36.197626 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.197584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:39:36.200310 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.200285 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 08 11:39:36.200405 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.200366 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrjk4\"" Jun 08 11:39:36.200405 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.200394 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 08 11:39:36.200496 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.200442 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 08 11:39:36.200725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.200709 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 08 11:39:36.200770 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.200733 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kpp27\"" Jun 08 11:39:36.206508 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.206483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cad1fdb-2d67-433d-96b9-b2f815229312-config-volume\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.206591 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.206517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/009f5259-e6a0-45d9-87e3-042f9313cf92-cert\") pod \"ingress-canary-r7kbj\" (UID: \"009f5259-e6a0-45d9-87e3-042f9313cf92\") " pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.206662 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.206600 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.206701 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.206671 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s76j\" (UniqueName: \"kubernetes.io/projected/4cad1fdb-2d67-433d-96b9-b2f815229312-kube-api-access-4s76j\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.206750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.206735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjwk\" (UniqueName: \"kubernetes.io/projected/009f5259-e6a0-45d9-87e3-042f9313cf92-kube-api-access-9xjwk\") pod \"ingress-canary-r7kbj\" (UID: \"009f5259-e6a0-45d9-87e3-042f9313cf92\") " pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.206784 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.206762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad1fdb-2d67-433d-96b9-b2f815229312-tmp-dir\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.268135 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.268101 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:36.307897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.307873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjwk\" (UniqueName: \"kubernetes.io/projected/009f5259-e6a0-45d9-87e3-042f9313cf92-kube-api-access-9xjwk\") pod \"ingress-canary-r7kbj\" (UID: \"009f5259-e6a0-45d9-87e3-042f9313cf92\") " pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.308027 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.307902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad1fdb-2d67-433d-96b9-b2f815229312-tmp-dir\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.308027 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.307944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cad1fdb-2d67-433d-96b9-b2f815229312-config-volume\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.308027 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.307961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/009f5259-e6a0-45d9-87e3-042f9313cf92-cert\") pod \"ingress-canary-r7kbj\" (UID: \"009f5259-e6a0-45d9-87e3-042f9313cf92\") " pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.308195 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.308129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.308195 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.308176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s76j\" (UniqueName: \"kubernetes.io/projected/4cad1fdb-2d67-433d-96b9-b2f815229312-kube-api-access-4s76j\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.308297 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.308222 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad1fdb-2d67-433d-96b9-b2f815229312-tmp-dir\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.308353 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:36.308328 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:39:36.308425 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:36.308411 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:36.808391448 +0000 UTC m=+36.110168124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:39:36.308566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.308547 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cad1fdb-2d67-433d-96b9-b2f815229312-config-volume\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.312183 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.312164 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/009f5259-e6a0-45d9-87e3-042f9313cf92-cert\") pod \"ingress-canary-r7kbj\" (UID: \"009f5259-e6a0-45d9-87e3-042f9313cf92\") " pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.318130 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.318103 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjwk\" (UniqueName: \"kubernetes.io/projected/009f5259-e6a0-45d9-87e3-042f9313cf92-kube-api-access-9xjwk\") pod \"ingress-canary-r7kbj\" (UID: \"009f5259-e6a0-45d9-87e3-042f9313cf92\") " pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.319316 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.319299 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s76j\" (UniqueName: \"kubernetes.io/projected/4cad1fdb-2d67-433d-96b9-b2f815229312-kube-api-access-4s76j\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.367437 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.367378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:36.411332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.411309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7kbj" Jun 08 11:39:36.555668 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.555631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:36.655567 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.655485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:36.811910 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:36.811873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:36.812091 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:36.812016 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:39:36.812091 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:36.812075 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:37.812058196 +0000 UTC m=+37.113834871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:39:37.820938 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:37.820903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:37.821353 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:37.821066 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:39:37.821353 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:37.821149 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:39.821133148 +0000 UTC m=+39.122909826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:39:38.109244 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:38.109171 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7kbj"] Jun 08 11:39:38.113847 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:39:38.113820 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009f5259_e6a0_45d9_87e3_042f9313cf92.slice/crio-bae2ef1613a4a0d0b50a6f85e1cbe3dc3da7bce1a5ec1e304eb73614f85bf501 WatchSource:0}: Error finding container bae2ef1613a4a0d0b50a6f85e1cbe3dc3da7bce1a5ec1e304eb73614f85bf501: Status 404 returned error can't find the container with id bae2ef1613a4a0d0b50a6f85e1cbe3dc3da7bce1a5ec1e304eb73614f85bf501 Jun 08 11:39:38.457811 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:38.457777 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:38.461270 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:38.461244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7kbj" event={"ID":"009f5259-e6a0-45d9-87e3-042f9313cf92","Type":"ContainerStarted","Data":"bae2ef1613a4a0d0b50a6f85e1cbe3dc3da7bce1a5ec1e304eb73614f85bf501"} Jun 08 11:39:38.558604 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:38.558569 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:39.467019 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:39.466988 2562 generic.go:356] "Generic (PLEG): container finished" podID="4933408b-be9c-45fc-ac36-622024c3d641" containerID="e4cb1955395c4ac07ed661e896d59a598be364961c3787a34702109d3fb129ed" exitCode=0 Jun 08 11:39:39.467515 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:39.467053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerDied","Data":"e4cb1955395c4ac07ed661e896d59a598be364961c3787a34702109d3fb129ed"} Jun 08 11:39:39.837832 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:39.837797 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:39.837956 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:39.837938 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:39:39.838013 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:39.838003 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:43.837987907 +0000 UTC m=+43.139764585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:39:40.300949 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:40.300908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:40.380484 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:40.380447 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:40.470386 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:40.470357 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7kbj" event={"ID":"009f5259-e6a0-45d9-87e3-042f9313cf92","Type":"ContainerStarted","Data":"1ff248cf05218043802ee6825141c1a1f177a778d04a24d6adb66390ea898320"} Jun 08 11:39:40.473439 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:40.473415 2562 generic.go:356] "Generic (PLEG): container finished" podID="4933408b-be9c-45fc-ac36-622024c3d641" containerID="9b01e5334574bf6e09fdbda0bc05c8e1c07e8975ff928280324d519b655fe0eb" exitCode=0 Jun 08 11:39:40.473555 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:40.473467 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerDied","Data":"9b01e5334574bf6e09fdbda0bc05c8e1c07e8975ff928280324d519b655fe0eb"} Jun 08 11:39:40.490000 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:40.489958 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r7kbj" podStartSLOduration=2.7009209800000002 podStartE2EDuration="4.489947337s" totalImagesPullingTime="1.789026357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:40.488910474 +0000 UTC m=+39.790687172" watchObservedRunningTime="2026-06-08 11:39:40.489947337 +0000 UTC m=+39.791724033" Jun 08 11:39:40.816210 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:40.816126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:41.185277 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:41.185198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:41.282117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:41.282078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:41.479014 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:41.478946 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" event={"ID":"4933408b-be9c-45fc-ac36-622024c3d641","Type":"ContainerStarted","Data":"b110c2fa52007395b4a9d9ffa935e9f65a72b3fbae169d049055e6d136e7fb35"} Jun 08 11:39:41.518361 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:41.518320 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bbxx9" podStartSLOduration=9.29523936 podStartE2EDuration="40.518298314s" totalImagesPullingTime="31.223058954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:39:41.511582858 +0000 UTC m=+40.813359556" watchObservedRunningTime="2026-06-08 11:39:41.518298314 +0000 UTC m=+40.820075010" Jun 08 11:39:41.665671 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:41.665632 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:41.757299 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:41.757232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:43.864496 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:43.864452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:43.864872 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:43.864605 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:39:43.864872 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:43.864688 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:39:51.864672171 +0000 UTC m=+51.166448845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:39:48.683694 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:48.683633 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:48.750511 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:48.750474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:39:51.918484 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:39:51.918449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:39:51.919143 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:51.918648 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:39:51.919143 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:39:51.918748 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:40:07.918723209 +0000 UTC m=+67.220499889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:40:02.378306 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:02.378272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:40:02.456038 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:02.455992 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xr47f" Jun 08 11:40:02.478139 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:02.478101 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:40:02.700355 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:02.700258 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:40:02.804363 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:02.804324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:40:06.924549 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:06.924497 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:40:06.924549 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:06.924551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:40:06.927234 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:06.927215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 08 11:40:06.927291 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:06.927273 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 08 11:40:06.935070 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:40:06.935049 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 08 11:40:06.935129 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:40:06.935119 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:41:10.935102871 +0000 UTC m=+130.236879547 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : secret "metrics-daemon-secret" not found Jun 08 11:40:06.938368 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:06.938341 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab-original-pull-secret\") pod \"global-pull-secret-syncer-tk87d\" (UID: \"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab\") " pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:40:07.025700 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.025655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:40:07.028413 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.028355 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 08 11:40:07.038020 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.037997 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 08 11:40:07.049233 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.049203 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6hq2\" (UniqueName: \"kubernetes.io/projected/8c5da3e0-9dfe-4570-9c0f-2e804419ed32-kube-api-access-n6hq2\") pod \"network-check-target-hxt4l\" (UID: \"8c5da3e0-9dfe-4570-9c0f-2e804419ed32\") " pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:40:07.106070 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.106034 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tk87d" Jun 08 11:40:07.113632 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.113600 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kpp27\"" Jun 08 11:40:07.122459 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.122430 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:40:07.254406 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.254373 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tk87d"] Jun 08 11:40:07.257243 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:40:07.257181 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec330fb8_a79b_4dab_a8c9_5be5d5dcb0ab.slice/crio-209a3dfec6d58c4f55d353d066dfc45caad106d024174880e19f3ae13435284d WatchSource:0}: Error finding container 209a3dfec6d58c4f55d353d066dfc45caad106d024174880e19f3ae13435284d: Status 404 returned error can't find the container with id 209a3dfec6d58c4f55d353d066dfc45caad106d024174880e19f3ae13435284d Jun 08 11:40:07.267454 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.267430 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxt4l"] Jun 08 11:40:07.269813 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:40:07.269793 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5da3e0_9dfe_4570_9c0f_2e804419ed32.slice/crio-afebea1139dd95afc0ba83e40dc7aee261bf16502a344c89ffe42778de3d93fe WatchSource:0}: Error finding container afebea1139dd95afc0ba83e40dc7aee261bf16502a344c89ffe42778de3d93fe: Status 404 returned error can't find the container with id afebea1139dd95afc0ba83e40dc7aee261bf16502a344c89ffe42778de3d93fe Jun 08 11:40:07.538952 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.538915 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxt4l" event={"ID":"8c5da3e0-9dfe-4570-9c0f-2e804419ed32","Type":"ContainerStarted","Data":"afebea1139dd95afc0ba83e40dc7aee261bf16502a344c89ffe42778de3d93fe"} Jun 08 11:40:07.540002 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.539975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tk87d" event={"ID":"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab","Type":"ContainerStarted","Data":"209a3dfec6d58c4f55d353d066dfc45caad106d024174880e19f3ae13435284d"} Jun 08 11:40:07.936572 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:07.936487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:40:07.936982 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:40:07.936633 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:40:07.936982 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:40:07.936707 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:40:39.936685225 +0000 UTC m=+99.238461981 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:40:12.555431 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:12.555391 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxt4l" event={"ID":"8c5da3e0-9dfe-4570-9c0f-2e804419ed32","Type":"ContainerStarted","Data":"31134e08873c895462e09e83b62e98474180be6c6ef0ff0035fec0123d6da0bc"} Jun 08 11:40:12.555880 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:12.555512 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:40:12.556957 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:12.556935 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tk87d" event={"ID":"ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab","Type":"ContainerStarted","Data":"ad2953555a3ca8f93dd1ad5182dec1537956b8ec6610ba647466c94a843b929a"} Jun 08 11:40:12.570770 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:12.570732 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hxt4l" podStartSLOduration=66.666822775 podStartE2EDuration="1m11.570719976s" totalImagesPullingTime="4.903897201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:40:12.569558608 +0000 UTC m=+71.871335305" watchObservedRunningTime="2026-06-08 11:40:12.570719976 +0000 UTC m=+71.872496673" Jun 08 11:40:12.582580 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:12.582542 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tk87d" podStartSLOduration=66.655547691 podStartE2EDuration="1m11.582530186s" totalImagesPullingTime="4.926982495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:40:12.582312025 +0000 UTC m=+71.884088723" watchObservedRunningTime="2026-06-08 11:40:12.582530186 +0000 UTC m=+71.884306883" Jun 08 11:40:15.399212 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.399180 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp"] Jun 08 11:40:15.401306 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.401291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.403604 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.403555 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jun 08 11:40:15.403763 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.403637 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jun 08 11:40:15.403814 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.403787 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jun 08 11:40:15.404656 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.404637 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jun 08 11:40:15.418030 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.418008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp"] Jun 08 11:40:15.486821 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.486790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0124487-1d8b-45e5-888b-2a4ba324adc8-tmp\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.486821 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.486823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c0124487-1d8b-45e5-888b-2a4ba324adc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.486997 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.486924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zms9\" (UniqueName: \"kubernetes.io/projected/c0124487-1d8b-45e5-888b-2a4ba324adc8-kube-api-access-2zms9\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.588017 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.587983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zms9\" (UniqueName: \"kubernetes.io/projected/c0124487-1d8b-45e5-888b-2a4ba324adc8-kube-api-access-2zms9\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.588213 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.588027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0124487-1d8b-45e5-888b-2a4ba324adc8-tmp\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.588213 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.588044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c0124487-1d8b-45e5-888b-2a4ba324adc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.588413 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.588394 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0124487-1d8b-45e5-888b-2a4ba324adc8-tmp\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.590557 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.590538 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c0124487-1d8b-45e5-888b-2a4ba324adc8-klusterlet-config\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.596826 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.596798 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zms9\" (UniqueName: \"kubernetes.io/projected/c0124487-1d8b-45e5-888b-2a4ba324adc8-kube-api-access-2zms9\") pod \"klusterlet-addon-workmgr-fd498d8f5-z6bdp\" (UID: \"c0124487-1d8b-45e5-888b-2a4ba324adc8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.710230 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.710143 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:15.829331 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:15.829301 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp"] Jun 08 11:40:15.833260 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:40:15.833231 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0124487_1d8b_45e5_888b_2a4ba324adc8.slice/crio-7fc979117e43c0c6798da9e6fd196c29b5e91981f7a2708ea9afce03bd97d52f WatchSource:0}: Error finding container 7fc979117e43c0c6798da9e6fd196c29b5e91981f7a2708ea9afce03bd97d52f: Status 404 returned error can't find the container with id 7fc979117e43c0c6798da9e6fd196c29b5e91981f7a2708ea9afce03bd97d52f Jun 08 11:40:16.566861 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:16.566810 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" event={"ID":"c0124487-1d8b-45e5-888b-2a4ba324adc8","Type":"ContainerStarted","Data":"7fc979117e43c0c6798da9e6fd196c29b5e91981f7a2708ea9afce03bd97d52f"} Jun 08 11:40:20.578559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:20.578516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" event={"ID":"c0124487-1d8b-45e5-888b-2a4ba324adc8","Type":"ContainerStarted","Data":"53c5a59fbe74cde4e5234523cb84f63ac88817361f117d70e6d55ffc8027bafc"} Jun 08 11:40:20.579044 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:20.578700 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:20.580475 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:20.580454 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:40:20.594856 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:20.594813 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" podStartSLOduration=1.5617523709999999 podStartE2EDuration="5.594798508s" totalImagesPullingTime="4.033046137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:40:20.593952047 +0000 UTC m=+79.895728745" watchObservedRunningTime="2026-06-08 11:40:20.594798508 +0000 UTC m=+79.896575183" Jun 08 11:40:39.951626 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:39.951576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:40:39.952022 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:40:39.951694 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:40:39.952022 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:40:39.951759 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:41:43.951745956 +0000 UTC m=+163.253522630 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:40:43.561743 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:40:43.561713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hxt4l" Jun 08 11:41:10.961517 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:41:10.961478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:41:10.961942 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:41:10.961654 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 08 11:41:10.961942 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:41:10.961737 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs podName:7283a8b3-5f6e-4dc0-addb-cee7c7299122 nodeName:}" failed. No retries permitted until 2026-06-08 11:43:12.961717632 +0000 UTC m=+252.263494310 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs") pod "network-metrics-daemon-6d2j9" (UID: "7283a8b3-5f6e-4dc0-addb-cee7c7299122") : secret "metrics-daemon-secret" not found Jun 08 11:41:39.088000 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:41:39.087941 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-74ptp" podUID="4cad1fdb-2d67-433d-96b9-b2f815229312" Jun 08 11:41:39.216319 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:41:39.216269 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6d2j9" podUID="7283a8b3-5f6e-4dc0-addb-cee7c7299122" Jun 08 11:41:39.772673 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:41:39.772640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74ptp" Jun 08 11:41:43.982668 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:41:43.982605 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:41:43.983103 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:41:43.982745 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 08 11:41:43.983103 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:41:43.982801 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls podName:4cad1fdb-2d67-433d-96b9-b2f815229312 nodeName:}" failed. No retries permitted until 2026-06-08 11:43:45.98278487 +0000 UTC m=+285.284561545 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls") pod "dns-default-74ptp" (UID: "4cad1fdb-2d67-433d-96b9-b2f815229312") : secret "dns-default-metrics-tls" not found Jun 08 11:41:52.197664 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:41:52.197527 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:42:17.236124 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:17.236096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:20.579812 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:20.579757 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" podUID="c0124487-1d8b-45e5-888b-2a4ba324adc8" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused" Jun 08 11:42:20.873482 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:20.873392 2562 generic.go:356] "Generic (PLEG): container finished" podID="c0124487-1d8b-45e5-888b-2a4ba324adc8" containerID="53c5a59fbe74cde4e5234523cb84f63ac88817361f117d70e6d55ffc8027bafc" exitCode=1 Jun 08 11:42:20.873482 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:20.873449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" event={"ID":"c0124487-1d8b-45e5-888b-2a4ba324adc8","Type":"ContainerDied","Data":"53c5a59fbe74cde4e5234523cb84f63ac88817361f117d70e6d55ffc8027bafc"} Jun 08 11:42:20.873809 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:20.873792 2562 scope.go:117] "RemoveContainer" containerID="53c5a59fbe74cde4e5234523cb84f63ac88817361f117d70e6d55ffc8027bafc" Jun 08 11:42:21.877947 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:21.877908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" event={"ID":"c0124487-1d8b-45e5-888b-2a4ba324adc8","Type":"ContainerStarted","Data":"920fb619a47d8df1cf331887fb5c44a5f2906e6ef82974b1ec6a3de4d7aa81c2"} Jun 08 11:42:21.878375 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:21.878181 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:42:21.878818 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:21.878801 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fd498d8f5-z6bdp" Jun 08 11:42:38.277377 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:38.277340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:40.166250 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.166209 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx"] Jun 08 11:42:40.169147 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.169126 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ff68ccc45-5hc4t"] Jun 08 11:42:40.169266 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.169244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.172160 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.172135 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.178390 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.178358 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jun 08 11:42:40.178658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.178645 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bpxx2\"" Jun 08 11:42:40.178797 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.178769 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:42:40.178867 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.178777 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jun 08 11:42:40.181068 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.181046 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jun 08 11:42:40.192186 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.192157 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx"] Jun 08 11:42:40.193071 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.193046 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jun 08 11:42:40.193227 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.193151 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jun 08 11:42:40.193397 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.193381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jun 08 11:42:40.195294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.195270 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pbtm8\"" Jun 08 11:42:40.202945 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.202913 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jun 08 11:42:40.220993 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.220959 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff68ccc45-5hc4t"] Jun 08 11:42:40.291422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-bound-sa-token\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.291422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291424 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbwp8\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-kube-api-access-bbwp8\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.291737 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aecca07d-03c6-4998-947c-a42108f7ac51-serving-cert\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.291808 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aecca07d-03c6-4998-947c-a42108f7ac51-config\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.291864 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-image-registry-private-configuration\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.291912 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b672ba9-0000-496d-a51e-14679cd37e2b-ca-trust-extracted\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.291958 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknm6\" (UniqueName: \"kubernetes.io/projected/aecca07d-03c6-4998-947c-a42108f7ac51-kube-api-access-sknm6\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.291958 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-installation-pull-secrets\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.292044 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.292044 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.291983 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-certificates\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.292044 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.292003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-trusted-ca\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.292595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.292565 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-896549976-4bxmm"] Jun 08 11:42:40.295396 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.295374 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw"] Jun 08 11:42:40.295519 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.295500 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.298290 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.298265 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c75c5578f-n574c"] Jun 08 11:42:40.298433 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.298365 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:40.299624 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.299587 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jun 08 11:42:40.299754 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.299659 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jun 08 11:42:40.301152 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.301134 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.306019 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.305989 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jun 08 11:42:40.306174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.306049 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jun 08 11:42:40.306174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.306049 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jun 08 11:42:40.306174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.306080 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xthnd\"" Jun 08 11:42:40.306174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.306002 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jun 08 11:42:40.306450 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.306435 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jun 08 11:42:40.306550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.306524 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-28tgt\"" Jun 08 11:42:40.314173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.314145 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2d4tx\"" Jun 08 11:42:40.315202 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.315178 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jun 08 11:42:40.315319 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.315177 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jun 08 11:42:40.315319 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.315178 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jun 08 11:42:40.315665 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.315594 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jun 08 11:42:40.315735 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.315719 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jun 08 11:42:40.315850 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.315836 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jun 08 11:42:40.318952 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.318930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-896549976-4bxmm"] Jun 08 11:42:40.322039 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.322017 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jun 08 11:42:40.327443 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.327410 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c75c5578f-n574c"] Jun 08 11:42:40.333266 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.333232 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw"] Jun 08 11:42:40.392940 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.392899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-bound-sa-token\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.393204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.392976 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknm6\" (UniqueName: \"kubernetes.io/projected/aecca07d-03c6-4998-947c-a42108f7ac51-kube-api-access-sknm6\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.393204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aecca07d-03c6-4998-947c-a42108f7ac51-serving-cert\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.393204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393170 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aecca07d-03c6-4998-947c-a42108f7ac51-config\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.393204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-image-registry-private-configuration\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.393410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-trusted-ca\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.393410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncqh\" (UniqueName: \"kubernetes.io/projected/98c1f2fb-d39c-4829-98aa-cf23f979f58e-kube-api-access-6ncqh\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:40.393410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a0e723-0c3c-4385-a198-ee3b4746fb94-trusted-ca-bundle\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.393410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-default-certificate\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.393602 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.393696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.393696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a0e723-0c3c-4385-a198-ee3b4746fb94-serving-cert\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.393795 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-stats-auth\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.393795 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a3a0e723-0c3c-4385-a198-ee3b4746fb94-snapshots\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.393795 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:40.393795 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbwp8\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-kube-api-access-bbwp8\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3a0e723-0c3c-4385-a198-ee3b4746fb94-tmp\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-installation-pull-secrets\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393884 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a0e723-0c3c-4385-a198-ee3b4746fb94-service-ca-bundle\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-certificates\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b672ba9-0000-496d-a51e-14679cd37e2b-ca-trust-extracted\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.393991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.394015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6d5d\" (UniqueName: \"kubernetes.io/projected/86fee2ac-8922-4a3c-890c-760527f5ea2f-kube-api-access-b6d5d\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.394154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.394042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxc9m\" (UniqueName: \"kubernetes.io/projected/a3a0e723-0c3c-4385-a198-ee3b4746fb94-kube-api-access-hxc9m\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.394981 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.394368 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 08 11:42:40.394981 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.394387 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff68ccc45-5hc4t: secret "image-registry-tls" not found Jun 08 11:42:40.394981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.394414 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b672ba9-0000-496d-a51e-14679cd37e2b-ca-trust-extracted\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.394981 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.394492 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls podName:7b672ba9-0000-496d-a51e-14679cd37e2b nodeName:}" failed. No retries permitted until 2026-06-08 11:42:40.894472936 +0000 UTC m=+220.196249626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls") pod "image-registry-6ff68ccc45-5hc4t" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b") : secret "image-registry-tls" not found Jun 08 11:42:40.394981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.394516 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-certificates\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.394981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.394545 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aecca07d-03c6-4998-947c-a42108f7ac51-config\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.394981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.394782 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-trusted-ca\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.395980 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.395954 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aecca07d-03c6-4998-947c-a42108f7ac51-serving-cert\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.396085 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.396040 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-image-registry-private-configuration\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.397003 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.396987 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-installation-pull-secrets\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.417741 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.417705 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-bound-sa-token\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.417902 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.417798 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbwp8\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-kube-api-access-bbwp8\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.417902 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.417802 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknm6\" (UniqueName: \"kubernetes.io/projected/aecca07d-03c6-4998-947c-a42108f7ac51-kube-api-access-sknm6\") pod \"service-ca-operator-6bc5ff68f5-lhmlx\" (UID: \"aecca07d-03c6-4998-947c-a42108f7ac51\") " pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.478598 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.478544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" Jun 08 11:42:40.495219 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6d5d\" (UniqueName: \"kubernetes.io/projected/86fee2ac-8922-4a3c-890c-760527f5ea2f-kube-api-access-b6d5d\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.495408 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxc9m\" (UniqueName: \"kubernetes.io/projected/a3a0e723-0c3c-4385-a198-ee3b4746fb94-kube-api-access-hxc9m\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.495408 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncqh\" (UniqueName: \"kubernetes.io/projected/98c1f2fb-d39c-4829-98aa-cf23f979f58e-kube-api-access-6ncqh\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:40.495408 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a0e723-0c3c-4385-a198-ee3b4746fb94-trusted-ca-bundle\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.495408 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-default-certificate\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.495408 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.495408 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495378 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.495408 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a0e723-0c3c-4385-a198-ee3b4746fb94-serving-cert\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-stats-auth\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a3a0e723-0c3c-4385-a198-ee3b4746fb94-snapshots\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3a0e723-0c3c-4385-a198-ee3b4746fb94-tmp\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.495544 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a0e723-0c3c-4385-a198-ee3b4746fb94-service-ca-bundle\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.495566 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:40.995543856 +0000 UTC m=+220.297320534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : configmap references non-existent config key: service-ca.crt Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.495699 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:42:40.495783 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.495737 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:40.995726631 +0000 UTC m=+220.297503305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : secret "router-metrics-certs-default" not found Jun 08 11:42:40.496145 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.495981 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:42:40.496145 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.496031 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls podName:98c1f2fb-d39c-4829-98aa-cf23f979f58e nodeName:}" failed. No retries permitted until 2026-06-08 11:42:40.996015804 +0000 UTC m=+220.297792481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dpfbw" (UID: "98c1f2fb-d39c-4829-98aa-cf23f979f58e") : secret "samples-operator-tls" not found Jun 08 11:42:40.496349 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.496318 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3a0e723-0c3c-4385-a198-ee3b4746fb94-tmp\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.496469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.496395 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a3a0e723-0c3c-4385-a198-ee3b4746fb94-snapshots\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.496811 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.496771 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a0e723-0c3c-4385-a198-ee3b4746fb94-trusted-ca-bundle\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.497213 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.497191 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3a0e723-0c3c-4385-a198-ee3b4746fb94-service-ca-bundle\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.498787 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.498752 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-stats-auth\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.498887 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.498803 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3a0e723-0c3c-4385-a198-ee3b4746fb94-serving-cert\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.498968 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.498950 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-default-certificate\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.516537 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.516509 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxc9m\" (UniqueName: \"kubernetes.io/projected/a3a0e723-0c3c-4385-a198-ee3b4746fb94-kube-api-access-hxc9m\") pod \"insights-operator-896549976-4bxmm\" (UID: \"a3a0e723-0c3c-4385-a198-ee3b4746fb94\") " pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.516807 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.516783 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6d5d\" (UniqueName: \"kubernetes.io/projected/86fee2ac-8922-4a3c-890c-760527f5ea2f-kube-api-access-b6d5d\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.518364 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.518346 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncqh\" (UniqueName: \"kubernetes.io/projected/98c1f2fb-d39c-4829-98aa-cf23f979f58e-kube-api-access-6ncqh\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:40.605776 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.605739 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-896549976-4bxmm" Jun 08 11:42:40.631375 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.631342 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx"] Jun 08 11:42:40.635600 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:42:40.635561 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecca07d_03c6_4998_947c_a42108f7ac51.slice/crio-937a8c21ebd24841ddac1e70b5f8f1a02aed074ec1f30b6c03e531775242f382 WatchSource:0}: Error finding container 937a8c21ebd24841ddac1e70b5f8f1a02aed074ec1f30b6c03e531775242f382: Status 404 returned error can't find the container with id 937a8c21ebd24841ddac1e70b5f8f1a02aed074ec1f30b6c03e531775242f382 Jun 08 11:42:40.725178 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.725140 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:40.784603 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.784566 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-896549976-4bxmm"] Jun 08 11:42:40.787481 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:42:40.787449 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a0e723_0c3c_4385_a198_ee3b4746fb94.slice/crio-c4484fecd686dc10305b15b6823dafa6497706412595d6ae188c50b01a8912c2 WatchSource:0}: Error finding container c4484fecd686dc10305b15b6823dafa6497706412595d6ae188c50b01a8912c2: Status 404 returned error can't find the container with id c4484fecd686dc10305b15b6823dafa6497706412595d6ae188c50b01a8912c2 Jun 08 11:42:40.894884 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.894795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:40.898849 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.898821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:40.898956 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.898936 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 08 11:42:40.898993 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.898958 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff68ccc45-5hc4t: secret "image-registry-tls" not found Jun 08 11:42:40.899028 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.899012 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls podName:7b672ba9-0000-496d-a51e-14679cd37e2b nodeName:}" failed. No retries permitted until 2026-06-08 11:42:41.898996556 +0000 UTC m=+221.200773232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls") pod "image-registry-6ff68ccc45-5hc4t" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b") : secret "image-registry-tls" not found Jun 08 11:42:40.935146 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.935103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" event={"ID":"aecca07d-03c6-4998-947c-a42108f7ac51","Type":"ContainerStarted","Data":"937a8c21ebd24841ddac1e70b5f8f1a02aed074ec1f30b6c03e531775242f382"} Jun 08 11:42:40.936176 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.936152 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-4bxmm" event={"ID":"a3a0e723-0c3c-4385-a198-ee3b4746fb94","Type":"ContainerStarted","Data":"c4484fecd686dc10305b15b6823dafa6497706412595d6ae188c50b01a8912c2"} Jun 08 11:42:40.999542 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.999494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.999749 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.999553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:40.999749 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:40.999583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:40.999749 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.999672 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:42:40.999749 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.999713 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:42:40.999749 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.999741 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:41.999724965 +0000 UTC m=+221.301501643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : secret "router-metrics-certs-default" not found Jun 08 11:42:40.999951 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.999757 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:41.999750185 +0000 UTC m=+221.301526860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : configmap references non-existent config key: service-ca.crt Jun 08 11:42:40.999951 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:40.999768 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls podName:98c1f2fb-d39c-4829-98aa-cf23f979f58e nodeName:}" failed. No retries permitted until 2026-06-08 11:42:41.999762665 +0000 UTC m=+221.301539340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dpfbw" (UID: "98c1f2fb-d39c-4829-98aa-cf23f979f58e") : secret "samples-operator-tls" not found Jun 08 11:42:41.907335 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:41.907302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:41.907781 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:41.907518 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 08 11:42:41.907781 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:41.907542 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff68ccc45-5hc4t: secret "image-registry-tls" not found Jun 08 11:42:41.907781 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:41.907636 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls podName:7b672ba9-0000-496d-a51e-14679cd37e2b nodeName:}" failed. No retries permitted until 2026-06-08 11:42:43.907594215 +0000 UTC m=+223.209370907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls") pod "image-registry-6ff68ccc45-5hc4t" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b") : secret "image-registry-tls" not found Jun 08 11:42:42.008250 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:42.008130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:42.008250 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:42.008195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:42.008250 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:42.008232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:42.008757 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:42.008318 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:42:42.008757 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:42.008369 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:42:42.008757 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:42.008400 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:44.008378258 +0000 UTC m=+223.310154948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : secret "router-metrics-certs-default" not found Jun 08 11:42:42.008757 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:42.008423 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls podName:98c1f2fb-d39c-4829-98aa-cf23f979f58e nodeName:}" failed. No retries permitted until 2026-06-08 11:42:44.00841372 +0000 UTC m=+223.310190398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dpfbw" (UID: "98c1f2fb-d39c-4829-98aa-cf23f979f58e") : secret "samples-operator-tls" not found Jun 08 11:42:42.008757 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:42.008437 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:44.008428754 +0000 UTC m=+223.310205429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : configmap references non-existent config key: service-ca.crt Jun 08 11:42:42.141250 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:42.141210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:42.241807 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:42.241714 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:43.925546 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:43.925511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:43.925966 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:43.925650 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 08 11:42:43.925966 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:43.925663 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff68ccc45-5hc4t: secret "image-registry-tls" not found Jun 08 11:42:43.925966 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:43.925715 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls podName:7b672ba9-0000-496d-a51e-14679cd37e2b nodeName:}" failed. No retries permitted until 2026-06-08 11:42:47.925700155 +0000 UTC m=+227.227476829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls") pod "image-registry-6ff68ccc45-5hc4t" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b") : secret "image-registry-tls" not found Jun 08 11:42:43.946507 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:43.946468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-4bxmm" event={"ID":"a3a0e723-0c3c-4385-a198-ee3b4746fb94","Type":"ContainerStarted","Data":"bca49ef23c32e43a6e9a5ef11a9d91cd80f2caffc230db88213ef7c2631028ac"} Jun 08 11:42:43.947900 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:43.947869 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" event={"ID":"aecca07d-03c6-4998-947c-a42108f7ac51","Type":"ContainerStarted","Data":"4d4206684be7a454a01a78b8e20adc39cc72d182313abb98de8f6e83a41d1604"} Jun 08 11:42:43.964055 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:43.963997 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-896549976-4bxmm" podStartSLOduration=1.801783788 podStartE2EDuration="3.963981586s" totalImagesPullingTime="2.162197798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:42:43.962402682 +0000 UTC m=+223.264179379" watchObservedRunningTime="2026-06-08 11:42:43.963981586 +0000 UTC m=+223.265758283" Jun 08 11:42:43.978723 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:43.978661 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" podStartSLOduration=1.661194644 podStartE2EDuration="3.978645035s" totalImagesPullingTime="2.317450391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:42:43.977933887 +0000 UTC m=+223.279710585" watchObservedRunningTime="2026-06-08 11:42:43.978645035 +0000 UTC m=+223.280421731" Jun 08 11:42:44.026476 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:44.026434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:44.026815 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:44.026790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:44.026892 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:44.026855 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:44.027123 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:44.026736 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:42:44.027179 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:44.026954 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:48.02692228 +0000 UTC m=+227.328698968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : configmap references non-existent config key: service-ca.crt Jun 08 11:42:44.027179 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:44.026971 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:42:44.027289 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:44.027185 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:48.027164873 +0000 UTC m=+227.328941550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : secret "router-metrics-certs-default" not found Jun 08 11:42:44.027289 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:44.027234 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls podName:98c1f2fb-d39c-4829-98aa-cf23f979f58e nodeName:}" failed. No retries permitted until 2026-06-08 11:42:48.027216397 +0000 UTC m=+227.328993074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dpfbw" (UID: "98c1f2fb-d39c-4829-98aa-cf23f979f58e") : secret "samples-operator-tls" not found Jun 08 11:42:44.759878 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:44.759826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:44.851550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:44.851508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:45.050045 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.050010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:45.203091 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.203052 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9"] Jun 08 11:42:45.206386 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.206364 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" Jun 08 11:42:45.209640 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.209594 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jun 08 11:42:45.209757 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.209645 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jun 08 11:42:45.210267 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.210249 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-74plq\"" Jun 08 11:42:45.263214 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.263183 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9"] Jun 08 11:42:45.293291 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.293257 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:45.338082 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.337982 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxrj\" (UniqueName: \"kubernetes.io/projected/924ac388-0c8f-42df-aa0a-5f33cd8f5f8d-kube-api-access-qbxrj\") pod \"migrator-955ffdf5f-6s8j9\" (UID: \"924ac388-0c8f-42df-aa0a-5f33cd8f5f8d\") " pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" Jun 08 11:42:45.393043 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.393010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:45.439420 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.439383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxrj\" (UniqueName: \"kubernetes.io/projected/924ac388-0c8f-42df-aa0a-5f33cd8f5f8d-kube-api-access-qbxrj\") pod \"migrator-955ffdf5f-6s8j9\" (UID: \"924ac388-0c8f-42df-aa0a-5f33cd8f5f8d\") " pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" Jun 08 11:42:45.461897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.461866 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxrj\" (UniqueName: \"kubernetes.io/projected/924ac388-0c8f-42df-aa0a-5f33cd8f5f8d-kube-api-access-qbxrj\") pod \"migrator-955ffdf5f-6s8j9\" (UID: \"924ac388-0c8f-42df-aa0a-5f33cd8f5f8d\") " pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" Jun 08 11:42:45.514901 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.514851 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" Jun 08 11:42:45.567694 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.567654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:45.654062 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.654014 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9"] Jun 08 11:42:45.658307 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:42:45.658267 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924ac388_0c8f_42df_aa0a_5f33cd8f5f8d.slice/crio-58856d09d57057c14f8e39bed7f9497eb8b923f937db486e730395775c4e44ea WatchSource:0}: Error finding container 58856d09d57057c14f8e39bed7f9497eb8b923f937db486e730395775c4e44ea: Status 404 returned error can't find the container with id 58856d09d57057c14f8e39bed7f9497eb8b923f937db486e730395775c4e44ea Jun 08 11:42:45.696432 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.696398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:45.955312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:45.955224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" event={"ID":"924ac388-0c8f-42df-aa0a-5f33cd8f5f8d","Type":"ContainerStarted","Data":"58856d09d57057c14f8e39bed7f9497eb8b923f937db486e730395775c4e44ea"} Jun 08 11:42:46.874761 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:46.874735 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qrh9w_96ad9462-64fc-482b-9657-4f5594b57e53/dns-node-resolver/0.log" Jun 08 11:42:46.960063 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:46.960027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" event={"ID":"924ac388-0c8f-42df-aa0a-5f33cd8f5f8d","Type":"ContainerStarted","Data":"e19ef1b99b4326c47b9331ece0433b1b289f7652c017809b6f0f1bff34a01662"} Jun 08 11:42:46.960343 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:46.960087 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" event={"ID":"924ac388-0c8f-42df-aa0a-5f33cd8f5f8d","Type":"ContainerStarted","Data":"59a478d7275eea4c609f025ed42bb6f4bbe0332b9adbf337b6e6c2fa062f1f5e"} Jun 08 11:42:46.976086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:46.976026 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-955ffdf5f-6s8j9" podStartSLOduration=0.88523097 podStartE2EDuration="1.976010758s" totalImagesPullingTime="1.090779788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:42:46.974814087 +0000 UTC m=+226.276590786" watchObservedRunningTime="2026-06-08 11:42:46.976010758 +0000 UTC m=+226.277787478" Jun 08 11:42:47.104913 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:47.104880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:47.202485 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:47.202450 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:47.665734 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:47.665706 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h6rx5_3b88db40-878a-4a55-a597-935a3b17f4fa/node-ca/0.log" Jun 08 11:42:47.959744 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:47.959654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:47.960116 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:47.959815 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jun 08 11:42:47.960116 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:47.959835 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff68ccc45-5hc4t: secret "image-registry-tls" not found Jun 08 11:42:47.960116 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:47.959889 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls podName:7b672ba9-0000-496d-a51e-14679cd37e2b nodeName:}" failed. No retries permitted until 2026-06-08 11:42:55.959871719 +0000 UTC m=+235.261648396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls") pod "image-registry-6ff68ccc45-5hc4t" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b") : secret "image-registry-tls" not found Jun 08 11:42:48.061077 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:48.061038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:48.061227 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:48.061089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:48.061227 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:48.061125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:48.061227 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:48.061194 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jun 08 11:42:48.061370 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:48.061258 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:56.061239732 +0000 UTC m=+235.363016410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : secret "router-metrics-certs-default" not found Jun 08 11:42:48.061370 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:48.061269 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jun 08 11:42:48.061370 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:48.061294 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:42:56.06127442 +0000 UTC m=+235.363051113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : configmap references non-existent config key: service-ca.crt Jun 08 11:42:48.061370 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:48.061321 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls podName:98c1f2fb-d39c-4829-98aa-cf23f979f58e nodeName:}" failed. No retries permitted until 2026-06-08 11:42:56.06131181 +0000 UTC m=+235.363088485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls") pod "cluster-samples-operator-854f8b96b6-dpfbw" (UID: "98c1f2fb-d39c-4829-98aa-cf23f979f58e") : secret "samples-operator-tls" not found Jun 08 11:42:48.069266 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:48.069236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:48.666750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:48.666717 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7kbj_009f5259-e6a0-45d9-87e3-042f9313cf92/serve-healthcheck-canary/0.log" Jun 08 11:42:50.197658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:50.197605 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:50.463090 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:50.463004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:50.563812 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:50.563769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:50.670908 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:50.670872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:50.772704 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:50.772596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:50.868550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:50.868518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:51.813548 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:51.813514 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:51.967979 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:51.967944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:52.275900 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:52.275819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:52.456023 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:52.455987 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:52.674196 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:52.674163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:52.873194 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:52.873158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:53.871573 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:53.871543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:53.981396 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:53.981361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:54.200626 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:54.200520 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:54.308985 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:54.308956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:54.497436 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:54.497354 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:54.691512 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:54.691471 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:54.883074 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:54.883029 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:55.904364 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:55.904328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:56.026976 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.026941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:56.029601 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.029576 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"image-registry-6ff68ccc45-5hc4t\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:56.072959 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.072924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:56.083859 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.083834 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:56.128343 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.128309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:56.128533 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.128419 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:56.128533 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.128445 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:56.128675 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:42:56.128626 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle podName:86fee2ac-8922-4a3c-890c-760527f5ea2f nodeName:}" failed. No retries permitted until 2026-06-08 11:43:12.12858888 +0000 UTC m=+251.430365575 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle") pod "router-default-6c75c5578f-n574c" (UID: "86fee2ac-8922-4a3c-890c-760527f5ea2f") : configmap references non-existent config key: service-ca.crt Jun 08 11:42:56.131531 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.131377 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86fee2ac-8922-4a3c-890c-760527f5ea2f-metrics-certs\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:42:56.131785 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.131756 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/98c1f2fb-d39c-4829-98aa-cf23f979f58e-samples-operator-tls\") pod \"cluster-samples-operator-854f8b96b6-dpfbw\" (UID: \"98c1f2fb-d39c-4829-98aa-cf23f979f58e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:56.212154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.212100 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" Jun 08 11:42:56.242240 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.240109 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff68ccc45-5hc4t"] Jun 08 11:42:56.356461 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.356432 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw"] Jun 08 11:42:56.457296 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.457210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:56.678681 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.678642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:56.834710 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.834679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:56.991274 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.991235 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" event={"ID":"7b672ba9-0000-496d-a51e-14679cd37e2b","Type":"ContainerStarted","Data":"17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925"} Jun 08 11:42:56.991792 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.991302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" event={"ID":"7b672ba9-0000-496d-a51e-14679cd37e2b","Type":"ContainerStarted","Data":"e7fbef2e7e4699f4718f87f9c24e7abf4c7fa90957a855f64711d7c97670e5a2"} Jun 08 11:42:56.991792 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.991351 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:42:56.992461 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:56.992438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" event={"ID":"98c1f2fb-d39c-4829-98aa-cf23f979f58e","Type":"ContainerStarted","Data":"b9d3fa22ae64fdedb5577542444cc61bf66846e1bfde1a8fa558c01ad9e86c72"} Jun 08 11:42:57.023001 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:57.022947 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" podStartSLOduration=17.022929227 podStartE2EDuration="17.022929227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:42:57.022214126 +0000 UTC m=+236.323990824" watchObservedRunningTime="2026-06-08 11:42:57.022929227 +0000 UTC m=+236.324705959" Jun 08 11:42:57.060418 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:57.060379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:57.440453 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:57.440417 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:57.778663 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:57.778527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:58.045966 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:58.045935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:58.449838 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:58.449717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:58.663538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:58.663498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:59.000603 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:59.000569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" event={"ID":"98c1f2fb-d39c-4829-98aa-cf23f979f58e","Type":"ContainerStarted","Data":"8f752d5d8b8eb9c68120930caeb367dac4ff595e838738fc0afbc0e30b5691f2"} Jun 08 11:42:59.000777 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:59.000636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" event={"ID":"98c1f2fb-d39c-4829-98aa-cf23f979f58e","Type":"ContainerStarted","Data":"6f22c8459de3ca9454ee66697aaa4635ec2d4bd064b346fc9f8c8c0896f9f8b7"} Jun 08 11:42:59.018053 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:59.017995 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-854f8b96b6-dpfbw" podStartSLOduration=16.963864593 podStartE2EDuration="19.017975806s" totalImagesPullingTime="2.054111213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:42:59.017135635 +0000 UTC m=+238.318912333" watchObservedRunningTime="2026-06-08 11:42:59.017975806 +0000 UTC m=+238.319752558" Jun 08 11:42:59.251887 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:59.251797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:59.644566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:59.644529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:42:59.834914 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:42:59.834870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:00.288924 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:00.288888 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:00.447363 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:00.447332 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:00.660013 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:00.659923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:00.881807 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:00.881769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:03.462180 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:03.462148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:03.563500 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:03.563472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:03.619696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:03.619664 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:03.776983 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:03.776901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:03.940842 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:03.940807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:04.003673 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:04.003641 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:04.342262 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:04.342233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:04.409835 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:04.409784 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:04.543114 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:04.543079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:04.604917 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:04.604838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:05.150955 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:05.150924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:05.351945 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:05.351908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:06.153480 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:06.153425 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:06.210714 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:06.210660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:08.449231 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:08.449197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:08.502578 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:08.502539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:09.454168 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.454132 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ff68ccc45-5hc4t"] Jun 08 11:43:09.458017 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.457978 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w5k7f"] Jun 08 11:43:09.464709 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.464679 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.468335 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.468306 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jun 08 11:43:09.468511 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.468307 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jun 08 11:43:09.468511 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.468307 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nrk4m\"" Jun 08 11:43:09.487556 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.487512 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w5k7f"] Jun 08 11:43:09.535867 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.535830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06aa3dd4-5f94-4804-8588-db2b94720738-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.536077 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.535910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd55m\" (UniqueName: \"kubernetes.io/projected/06aa3dd4-5f94-4804-8588-db2b94720738-kube-api-access-wd55m\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.536077 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.535941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06aa3dd4-5f94-4804-8588-db2b94720738-crio-socket\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.536077 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.536011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06aa3dd4-5f94-4804-8588-db2b94720738-data-volume\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.536077 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.536048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06aa3dd4-5f94-4804-8588-db2b94720738-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.594317 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.594285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:09.636519 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.636475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06aa3dd4-5f94-4804-8588-db2b94720738-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.636519 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.636521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd55m\" (UniqueName: \"kubernetes.io/projected/06aa3dd4-5f94-4804-8588-db2b94720738-kube-api-access-wd55m\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.636813 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.636550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06aa3dd4-5f94-4804-8588-db2b94720738-crio-socket\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.636813 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.636588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06aa3dd4-5f94-4804-8588-db2b94720738-data-volume\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.636813 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.636632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06aa3dd4-5f94-4804-8588-db2b94720738-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.636813 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.636687 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06aa3dd4-5f94-4804-8588-db2b94720738-crio-socket\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.637055 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.637029 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06aa3dd4-5f94-4804-8588-db2b94720738-data-volume\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.637191 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.637169 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06aa3dd4-5f94-4804-8588-db2b94720738-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.639284 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.639263 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06aa3dd4-5f94-4804-8588-db2b94720738-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.645522 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.645491 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd55m\" (UniqueName: \"kubernetes.io/projected/06aa3dd4-5f94-4804-8588-db2b94720738-kube-api-access-wd55m\") pod \"insights-runtime-extractor-w5k7f\" (UID: \"06aa3dd4-5f94-4804-8588-db2b94720738\") " pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.679651 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.679593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:09.774879 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.774778 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w5k7f" Jun 08 11:43:09.914893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:09.914859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w5k7f"] Jun 08 11:43:09.918966 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:09.918939 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06aa3dd4_5f94_4804_8588_db2b94720738.slice/crio-6f78b37a517d787aadf8b91bb415b68b164c3ca62b3795955a02360acf6f2364 WatchSource:0}: Error finding container 6f78b37a517d787aadf8b91bb415b68b164c3ca62b3795955a02360acf6f2364: Status 404 returned error can't find the container with id 6f78b37a517d787aadf8b91bb415b68b164c3ca62b3795955a02360acf6f2364 Jun 08 11:43:10.037320 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:10.037282 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5k7f" event={"ID":"06aa3dd4-5f94-4804-8588-db2b94720738","Type":"ContainerStarted","Data":"ac313e7d993ae32897398bdad041bb9f6f0d0a50597f090ca6ee15a4d5200c3b"} Jun 08 11:43:10.037513 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:10.037337 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5k7f" event={"ID":"06aa3dd4-5f94-4804-8588-db2b94720738","Type":"ContainerStarted","Data":"6f78b37a517d787aadf8b91bb415b68b164c3ca62b3795955a02360acf6f2364"} Jun 08 11:43:10.568498 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:10.568428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:10.671273 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:10.671241 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:11.045938 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:11.045896 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5k7f" event={"ID":"06aa3dd4-5f94-4804-8588-db2b94720738","Type":"ContainerStarted","Data":"4901e94f0f1ce7dd6e1da6420846eeffc37a35837b3b427964020fd8cab29fac"} Jun 08 11:43:12.159565 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:12.159520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:43:12.160128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:12.160108 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86fee2ac-8922-4a3c-890c-760527f5ea2f-service-ca-bundle\") pod \"router-default-6c75c5578f-n574c\" (UID: \"86fee2ac-8922-4a3c-890c-760527f5ea2f\") " pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:43:12.419582 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:12.419490 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-28tgt\"" Jun 08 11:43:12.428409 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:12.428380 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:43:12.586957 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:12.586925 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c75c5578f-n574c"] Jun 08 11:43:12.589862 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:12.589833 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fee2ac_8922_4a3c_890c_760527f5ea2f.slice/crio-29ab0b4327c6048dbcbca1565eb7358c87deae05816211d6fda4334822cc10c4 WatchSource:0}: Error finding container 29ab0b4327c6048dbcbca1565eb7358c87deae05816211d6fda4334822cc10c4: Status 404 returned error can't find the container with id 29ab0b4327c6048dbcbca1565eb7358c87deae05816211d6fda4334822cc10c4 Jun 08 11:43:12.966143 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:12.966046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:43:12.968636 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:12.968589 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7283a8b3-5f6e-4dc0-addb-cee7c7299122-metrics-certs\") pod \"network-metrics-daemon-6d2j9\" (UID: \"7283a8b3-5f6e-4dc0-addb-cee7c7299122\") " pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:43:13.056277 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.056236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w5k7f" event={"ID":"06aa3dd4-5f94-4804-8588-db2b94720738","Type":"ContainerStarted","Data":"80a9b9a5fa8dbb51094fb59ab7ec53f4e892acdda52cf39bba53d0e6087f1ac7"} Jun 08 11:43:13.057800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.057767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c75c5578f-n574c" event={"ID":"86fee2ac-8922-4a3c-890c-760527f5ea2f","Type":"ContainerStarted","Data":"01f03088fd5f7d7e6783686871a6baa77d32b88c06d95e4012c0e680c7aadf8d"} Jun 08 11:43:13.057949 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.057820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c75c5578f-n574c" event={"ID":"86fee2ac-8922-4a3c-890c-760527f5ea2f","Type":"ContainerStarted","Data":"29ab0b4327c6048dbcbca1565eb7358c87deae05816211d6fda4334822cc10c4"} Jun 08 11:43:13.084595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.084547 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w5k7f" podStartSLOduration=1.949968691 podStartE2EDuration="4.084530467s" totalImagesPullingTime="2.134561776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 11:43:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:13.08313623 +0000 UTC m=+252.384912926" watchObservedRunningTime="2026-06-08 11:43:13.084530467 +0000 UTC m=+252.386307163" Jun 08 11:43:13.118189 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.118132 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c75c5578f-n574c" podStartSLOduration=33.118113498 podStartE2EDuration="33.118113498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:13.117261007 +0000 UTC m=+252.419037703" watchObservedRunningTime="2026-06-08 11:43:13.118113498 +0000 UTC m=+252.419890191" Jun 08 11:43:13.201107 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.201064 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mrjk4\"" Jun 08 11:43:13.209008 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.208973 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6d2j9" Jun 08 11:43:13.364957 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.364921 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6d2j9"] Jun 08 11:43:13.368781 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:13.368745 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7283a8b3_5f6e_4dc0_addb_cee7c7299122.slice/crio-5d059fb09f5264856ecba051602055f2c5fa08a5892d18d65f449c74cb67728b WatchSource:0}: Error finding container 5d059fb09f5264856ecba051602055f2c5fa08a5892d18d65f449c74cb67728b: Status 404 returned error can't find the container with id 5d059fb09f5264856ecba051602055f2c5fa08a5892d18d65f449c74cb67728b Jun 08 11:43:13.429116 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.429078 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:43:13.432045 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:13.432020 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:43:14.063243 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:14.063205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6d2j9" event={"ID":"7283a8b3-5f6e-4dc0-addb-cee7c7299122","Type":"ContainerStarted","Data":"5d059fb09f5264856ecba051602055f2c5fa08a5892d18d65f449c74cb67728b"} Jun 08 11:43:14.063646 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:14.063603 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:43:14.064902 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:14.064712 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c75c5578f-n574c" Jun 08 11:43:14.590454 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:14.590373 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:14.696861 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:14.696825 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:14.767854 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:14.767815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:15.068817 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.068782 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6d2j9" event={"ID":"7283a8b3-5f6e-4dc0-addb-cee7c7299122","Type":"ContainerStarted","Data":"1cf066e0536b6482e20020d262d65c322420e3b429fa229a707b87b7be4d9669"} Jun 08 11:43:15.069064 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.068830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6d2j9" event={"ID":"7283a8b3-5f6e-4dc0-addb-cee7c7299122","Type":"ContainerStarted","Data":"0788c17ec91c152fa78c0dfd5d91ee7885a1b69be42408078f2f94b5f3e99eb1"} Jun 08 11:43:15.087495 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.087441 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6d2j9" podStartSLOduration=253.157824921 podStartE2EDuration="4m14.087425007s" totalImagesPullingTime="929.600086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:15.085585476 +0000 UTC m=+254.387362175" watchObservedRunningTime="2026-06-08 11:43:15.087425007 +0000 UTC m=+254.389201704" Jun 08 11:43:15.158630 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.158578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:15.259893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.259855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:15.360366 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.360281 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:15.552847 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.552817 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:15.617857 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:15.617784 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:16.555800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:16.555764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:16.949546 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:16.949462 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:17.050589 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:17.050549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:17.759676 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:17.759641 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:17.822081 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:17.822047 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:18.522798 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:18.522761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:19.460967 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.460926 2562 patch_prober.go:28] interesting pod/image-registry-6ff68ccc45-5hc4t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:43:19.461169 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.460985 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" podUID="7b672ba9-0000-496d-a51e-14679cd37e2b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:43:19.810056 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.810024 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd4cb8d68-cp9cw"] Jun 08 11:43:19.812944 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.812925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:19.815294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.815269 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jun 08 11:43:19.815484 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.815278 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jun 08 11:43:19.816094 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.816075 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jun 08 11:43:19.816222 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.816112 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jun 08 11:43:19.816222 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.816122 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jun 08 11:43:19.816222 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.816180 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jun 08 11:43:19.816395 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.816205 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rtp72\"" Jun 08 11:43:19.816395 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.816135 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jun 08 11:43:19.822053 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.822026 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd4cb8d68-cp9cw"] Jun 08 11:43:19.925988 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.925950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-serving-cert\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:19.925988 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.926007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-oauth-config\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:19.926261 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.926081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-oauth-serving-cert\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:19.926261 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.926117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-service-ca\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:19.926261 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.926143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992c9\" (UniqueName: \"kubernetes.io/projected/fb5f7442-3594-46d2-9dc2-9928cf6016eb-kube-api-access-992c9\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:19.926261 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:19.926169 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-config\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.004765 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.004727 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:20.026787 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.026751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-serving-cert\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.026891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.026809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-oauth-config\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.026891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.026837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-oauth-serving-cert\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.026891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.026854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-service-ca\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.026891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.026870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992c9\" (UniqueName: \"kubernetes.io/projected/fb5f7442-3594-46d2-9dc2-9928cf6016eb-kube-api-access-992c9\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.026891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.026889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-config\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.027650 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.027578 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-config\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.027752 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.027717 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-oauth-serving-cert\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.027752 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.027733 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-service-ca\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.029337 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.029306 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-oauth-config\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.029504 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.029482 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-serving-cert\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.035057 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.035030 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992c9\" (UniqueName: \"kubernetes.io/projected/fb5f7442-3594-46d2-9dc2-9928cf6016eb-kube-api-access-992c9\") pod \"console-6fd4cb8d68-cp9cw\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.107333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.107248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:20.122283 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.122242 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:20.255164 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.255122 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd4cb8d68-cp9cw"] Jun 08 11:43:20.259553 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:20.259516 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5f7442_3594_46d2_9dc2_9928cf6016eb.slice/crio-c529680fccfb674193c7305e1a48cdb5ced6319250d2626271df635592629090 WatchSource:0}: Error finding container c529680fccfb674193c7305e1a48cdb5ced6319250d2626271df635592629090: Status 404 returned error can't find the container with id c529680fccfb674193c7305e1a48cdb5ced6319250d2626271df635592629090 Jun 08 11:43:20.298421 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:20.298391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:21.092231 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:21.092195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd4cb8d68-cp9cw" event={"ID":"fb5f7442-3594-46d2-9dc2-9928cf6016eb","Type":"ContainerStarted","Data":"c529680fccfb674193c7305e1a48cdb5ced6319250d2626271df635592629090"} Jun 08 11:43:21.775058 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:21.775018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:21.876433 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:21.876403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:21.971845 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:21.971816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:22.100907 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.100871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:22.157922 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.157887 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zwvdw"] Jun 08 11:43:22.161336 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.161310 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.163981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.163957 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jun 08 11:43:22.164494 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.164373 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jun 08 11:43:22.164494 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.164384 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jun 08 11:43:22.164494 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.164404 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jun 08 11:43:22.164494 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.164427 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cpjq7\"" Jun 08 11:43:22.165047 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.165026 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jun 08 11:43:22.165154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.165129 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jun 08 11:43:22.192361 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.192325 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:22.246918 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.246867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ssf\" (UniqueName: \"kubernetes.io/projected/348aafc2-831c-480e-9d06-922f732a8754-kube-api-access-c5ssf\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.246918 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.246915 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-textfile\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.247173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.246939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/348aafc2-831c-480e-9d06-922f732a8754-metrics-client-ca\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.247173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.246964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-accelerators-collector-config\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.247173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.246989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-tls\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.247173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.247018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-sys\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.247173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.247044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.247173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.247091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-root\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.247489 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.247221 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-wtmp\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348541 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-sys\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348541 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348541 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348466 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-sys\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348541 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-root\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348634 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-root\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-wtmp\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ssf\" (UniqueName: \"kubernetes.io/projected/348aafc2-831c-480e-9d06-922f732a8754-kube-api-access-c5ssf\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-textfile\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348767 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-wtmp\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/348aafc2-831c-480e-9d06-922f732a8754-metrics-client-ca\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-accelerators-collector-config\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.348876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.348837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-tls\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.349230 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:43:22.348946 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jun 08 11:43:22.349230 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:43:22.349044 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-tls podName:348aafc2-831c-480e-9d06-922f732a8754 nodeName:}" failed. No retries permitted until 2026-06-08 11:43:22.849025382 +0000 UTC m=+262.150802057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-tls") pod "node-exporter-zwvdw" (UID: "348aafc2-831c-480e-9d06-922f732a8754") : secret "node-exporter-tls" not found Jun 08 11:43:22.349230 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.349207 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-textfile\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.349448 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.349382 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/348aafc2-831c-480e-9d06-922f732a8754-metrics-client-ca\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.349448 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.349442 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-accelerators-collector-config\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.351579 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.351507 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.362879 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.362834 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ssf\" (UniqueName: \"kubernetes.io/projected/348aafc2-831c-480e-9d06-922f732a8754-kube-api-access-c5ssf\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.370481 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.370433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:22.469184 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.469144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:22.568559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.568520 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:22.770159 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.770072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:22.852468 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.852415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-tls\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.855342 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.855313 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/348aafc2-831c-480e-9d06-922f732a8754-node-exporter-tls\") pod \"node-exporter-zwvdw\" (UID: \"348aafc2-831c-480e-9d06-922f732a8754\") " pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:22.892014 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:22.891977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:23.072827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.072792 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zwvdw" Jun 08 11:43:23.083571 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:23.083522 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348aafc2_831c_480e_9d06_922f732a8754.slice/crio-241ede3c2fc5a731a0ae6cbe9fd370ebc234c3bc8e095e2eb28eb61a227511d3 WatchSource:0}: Error finding container 241ede3c2fc5a731a0ae6cbe9fd370ebc234c3bc8e095e2eb28eb61a227511d3: Status 404 returned error can't find the container with id 241ede3c2fc5a731a0ae6cbe9fd370ebc234c3bc8e095e2eb28eb61a227511d3 Jun 08 11:43:23.100983 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.100948 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwvdw" event={"ID":"348aafc2-831c-480e-9d06-922f732a8754","Type":"ContainerStarted","Data":"241ede3c2fc5a731a0ae6cbe9fd370ebc234c3bc8e095e2eb28eb61a227511d3"} Jun 08 11:43:23.124383 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.124354 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:23.198924 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.198894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:23.203860 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.203828 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:43:23.207288 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.207264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.209816 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.209791 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 08 11:43:23.209938 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.209791 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 08 11:43:23.209938 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.209794 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 08 11:43:23.210110 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.210092 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z2g5g\"" Jun 08 11:43:23.210162 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.210111 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 08 11:43:23.210162 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.210095 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 08 11:43:23.210162 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.210145 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 08 11:43:23.210305 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.210205 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 08 11:43:23.210305 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.210240 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 08 11:43:23.210451 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.210435 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 08 11:43:23.223908 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.223878 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:43:23.256231 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256231 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256231 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-out\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256261 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256344 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-web-config\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgcj\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-kube-api-access-vvgcj\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256515 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256563 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256740 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256639 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.256740 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.256662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.357941 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.357856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.357941 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.357893 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.357941 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.357922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.357941 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.357938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-web-config\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358249 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.357958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358249 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.357978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgcj\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-kube-api-access-vvgcj\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358249 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358249 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:43:23.358049 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle podName:43b576bc-41ae-49de-8b23-ff6f5e262cac nodeName:}" failed. No retries permitted until 2026-06-08 11:43:23.858023895 +0000 UTC m=+263.159800585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac") : configmap references non-existent config key: ca-bundle.crt Jun 08 11:43:23.358249 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358249 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358534 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358677 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.358677 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-out\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.359196 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.358983 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.360041 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.360014 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.361438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.361409 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.361906 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.361880 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-web-config\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.361998 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.361923 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.362059 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.362043 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.362306 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.362284 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-out\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.362424 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.362404 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-volume\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.362585 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.362559 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.362749 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.362728 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.363309 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.363287 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.376298 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.376272 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgcj\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-kube-api-access-vvgcj\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.436154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.436112 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:23.861669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.861628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:23.862350 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:23.862328 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:24.106011 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:24.105975 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd4cb8d68-cp9cw" event={"ID":"fb5f7442-3594-46d2-9dc2-9928cf6016eb","Type":"ContainerStarted","Data":"eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37"} Jun 08 11:43:24.116309 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:24.116247 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:43:24.134682 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:24.134585 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd4cb8d68-cp9cw" podStartSLOduration=2.312021882 podStartE2EDuration="5.134547526s" totalImagesPullingTime="2.822525644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:24.134142116 +0000 UTC m=+263.435918814" watchObservedRunningTime="2026-06-08 11:43:24.134547526 +0000 UTC m=+263.436324225" Jun 08 11:43:24.197806 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:24.197675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:24.258631 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:24.258585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:24.268501 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:24.268374 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:43:24.287598 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:24.287557 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b576bc_41ae_49de_8b23_ff6f5e262cac.slice/crio-0f355d890df82ad011685cb5173f60d673a3cd04648a91cdd54c36fff4c6c1a2 WatchSource:0}: Error finding container 0f355d890df82ad011685cb5173f60d673a3cd04648a91cdd54c36fff4c6c1a2: Status 404 returned error can't find the container with id 0f355d890df82ad011685cb5173f60d673a3cd04648a91cdd54c36fff4c6c1a2 Jun 08 11:43:25.090376 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.090325 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:25.111960 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.111917 2562 generic.go:356] "Generic (PLEG): container finished" podID="348aafc2-831c-480e-9d06-922f732a8754" containerID="e906bcda20fb99c1d9fbb48d4f825f1035e0725a14043f03da7e1ec0060539ef" exitCode=0 Jun 08 11:43:25.112393 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.112045 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwvdw" event={"ID":"348aafc2-831c-480e-9d06-922f732a8754","Type":"ContainerDied","Data":"e906bcda20fb99c1d9fbb48d4f825f1035e0725a14043f03da7e1ec0060539ef"} Jun 08 11:43:25.113672 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.113629 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerStarted","Data":"0f355d890df82ad011685cb5173f60d673a3cd04648a91cdd54c36fff4c6c1a2"} Jun 08 11:43:25.169450 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.169411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:25.233837 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.233811 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:25.547596 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.547563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:25.610129 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:25.610048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:26.119299 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.119267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwvdw" event={"ID":"348aafc2-831c-480e-9d06-922f732a8754","Type":"ContainerStarted","Data":"237fef1c5046b4869875c0891745fcb5c88313e050bf5dd002a7aa4390e2b862"} Jun 08 11:43:26.119767 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.119329 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwvdw" event={"ID":"348aafc2-831c-480e-9d06-922f732a8754","Type":"ContainerStarted","Data":"7face463be98fe2ddbfa8b0b9d5f133e99c76ea9d8ceba5f24713023cbb61ebc"} Jun 08 11:43:26.120831 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.120808 2562 generic.go:356] "Generic (PLEG): container finished" podID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerID="c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510" exitCode=0 Jun 08 11:43:26.120919 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.120869 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510"} Jun 08 11:43:26.163980 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.163924 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zwvdw" podStartSLOduration=3.094394611 podStartE2EDuration="4.163909059s" totalImagesPullingTime="984.694936ms" totalInitContainerRuntime="84.819512ms" isStatefulPod=true podCreationTimestamp="2026-06-08 11:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:26.137774069 +0000 UTC m=+265.439550767" watchObservedRunningTime="2026-06-08 11:43:26.163909059 +0000 UTC m=+265.465685755" Jun 08 11:43:26.944806 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.944767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:26.945430 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.945402 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd4cb8d68-cp9cw"] Jun 08 11:43:26.949668 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.949638 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8"] Jun 08 11:43:26.953104 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.953084 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:26.955116 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.955095 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jun 08 11:43:26.955276 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.955256 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bcjvn\"" Jun 08 11:43:26.960865 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:26.960844 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8"] Jun 08 11:43:27.036458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.036415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:27.091832 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.091800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kp7\" (UniqueName: \"kubernetes.io/projected/f9717e3f-f320-4672-87f2-c5fb0afff294-kube-api-access-55kp7\") pod \"monitoring-plugin-76b68cc874-hcnc8\" (UID: \"f9717e3f-f320-4672-87f2-c5fb0afff294\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:27.092017 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.091859 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9717e3f-f320-4672-87f2-c5fb0afff294-monitoring-plugin-cert\") pod \"monitoring-plugin-76b68cc874-hcnc8\" (UID: \"f9717e3f-f320-4672-87f2-c5fb0afff294\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:27.183155 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.183111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:27.198837 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.193813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9717e3f-f320-4672-87f2-c5fb0afff294-monitoring-plugin-cert\") pod \"monitoring-plugin-76b68cc874-hcnc8\" (UID: \"f9717e3f-f320-4672-87f2-c5fb0afff294\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:27.198837 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.194030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kp7\" (UniqueName: \"kubernetes.io/projected/f9717e3f-f320-4672-87f2-c5fb0afff294-kube-api-access-55kp7\") pod \"monitoring-plugin-76b68cc874-hcnc8\" (UID: \"f9717e3f-f320-4672-87f2-c5fb0afff294\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:27.199315 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.199280 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f9717e3f-f320-4672-87f2-c5fb0afff294-monitoring-plugin-cert\") pod \"monitoring-plugin-76b68cc874-hcnc8\" (UID: \"f9717e3f-f320-4672-87f2-c5fb0afff294\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:27.202750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.202720 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kp7\" (UniqueName: \"kubernetes.io/projected/f9717e3f-f320-4672-87f2-c5fb0afff294-kube-api-access-55kp7\") pod \"monitoring-plugin-76b68cc874-hcnc8\" (UID: \"f9717e3f-f320-4672-87f2-c5fb0afff294\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:27.264684 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.264643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:27.649940 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:27.649911 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8"] Jun 08 11:43:27.652867 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:27.652841 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9717e3f_f320_4672_87f2_c5fb0afff294.slice/crio-0af375120aa6ba7552e3075b0c89f12247efcd64dcae2575509d8bfb87e97044 WatchSource:0}: Error finding container 0af375120aa6ba7552e3075b0c89f12247efcd64dcae2575509d8bfb87e97044: Status 404 returned error can't find the container with id 0af375120aa6ba7552e3075b0c89f12247efcd64dcae2575509d8bfb87e97044 Jun 08 11:43:28.130668 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:28.130632 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" event={"ID":"f9717e3f-f320-4672-87f2-c5fb0afff294","Type":"ContainerStarted","Data":"0af375120aa6ba7552e3075b0c89f12247efcd64dcae2575509d8bfb87e97044"} Jun 08 11:43:28.133290 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:28.133267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerStarted","Data":"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26"} Jun 08 11:43:28.133415 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:28.133333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerStarted","Data":"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8"} Jun 08 11:43:28.133415 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:28.133374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerStarted","Data":"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6"} Jun 08 11:43:28.133415 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:28.133412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerStarted","Data":"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103"} Jun 08 11:43:28.133546 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:28.133452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerStarted","Data":"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0"} Jun 08 11:43:28.380640 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:28.380574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:29.139511 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:29.139478 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerStarted","Data":"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729"} Jun 08 11:43:29.165657 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:29.165583 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.136640651 podStartE2EDuration="6.165564204s" totalImagesPullingTime="3.011751677s" totalInitContainerRuntime="17.171876ms" isStatefulPod=false podCreationTimestamp="2026-06-08 11:43:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:29.164870259 +0000 UTC m=+268.466646989" watchObservedRunningTime="2026-06-08 11:43:29.165564204 +0000 UTC m=+268.467340902" Jun 08 11:43:29.458650 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:29.458526 2562 patch_prober.go:28] interesting pod/image-registry-6ff68ccc45-5hc4t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jun 08 11:43:29.458650 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:29.458592 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" podUID="7b672ba9-0000-496d-a51e-14679cd37e2b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jun 08 11:43:29.581420 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:29.581389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:29.684843 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:29.684813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:30.122708 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:30.122671 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:30.144922 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:30.144889 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" event={"ID":"f9717e3f-f320-4672-87f2-c5fb0afff294","Type":"ContainerStarted","Data":"a884dc294e6c7377de85c23390b92cef2036d8225ba91b880ff46483b250041c"} Jun 08 11:43:30.145406 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:30.145386 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:30.149887 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:30.149865 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" Jun 08 11:43:30.161540 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:30.161497 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-76b68cc874-hcnc8" podStartSLOduration=2.713119237 podStartE2EDuration="4.161482774s" totalImagesPullingTime="1.448363537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:30.160776497 +0000 UTC m=+269.462553194" watchObservedRunningTime="2026-06-08 11:43:30.161482774 +0000 UTC m=+269.463259471" Jun 08 11:43:30.583837 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:30.583794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:30.687429 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:30.687399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:34.477071 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.477029 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" podUID="7b672ba9-0000-496d-a51e-14679cd37e2b" containerName="registry" containerID="cri-o://17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925" gracePeriod=30 Jun 08 11:43:34.717419 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.717394 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:43:34.859952 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.859916 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-certificates\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860142 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.859963 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-image-registry-private-configuration\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860142 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.859994 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-bound-sa-token\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860142 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.860018 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860142 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.860096 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b672ba9-0000-496d-a51e-14679cd37e2b-ca-trust-extracted\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860142 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.860119 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-trusted-ca\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860390 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.860151 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbwp8\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-kube-api-access-bbwp8\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860390 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.860175 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-installation-pull-secrets\") pod \"7b672ba9-0000-496d-a51e-14679cd37e2b\" (UID: \"7b672ba9-0000-496d-a51e-14679cd37e2b\") " Jun 08 11:43:34.860547 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.860510 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:43:34.860702 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.860583 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:43:34.862601 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.862537 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-kube-api-access-bbwp8" (OuterVolumeSpecName: "kube-api-access-bbwp8") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "kube-api-access-bbwp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:43:34.862937 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.862909 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:43:34.863049 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.863031 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:43:34.863104 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.863087 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:43:34.863141 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.863118 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:43:34.868208 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.868186 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b672ba9-0000-496d-a51e-14679cd37e2b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7b672ba9-0000-496d-a51e-14679cd37e2b" (UID: "7b672ba9-0000-496d-a51e-14679cd37e2b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:43:34.961740 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961689 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b672ba9-0000-496d-a51e-14679cd37e2b-ca-trust-extracted\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:34.961740 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961737 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-trusted-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:34.961740 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961748 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbwp8\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-kube-api-access-bbwp8\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:34.961740 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961758 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-installation-pull-secrets\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:34.961996 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961767 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-certificates\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:34.961996 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961778 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b672ba9-0000-496d-a51e-14679cd37e2b-image-registry-private-configuration\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:34.961996 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961787 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-bound-sa-token\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:34.961996 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:34.961795 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b672ba9-0000-496d-a51e-14679cd37e2b-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:35.163841 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.163753 2562 generic.go:356] "Generic (PLEG): container finished" podID="7b672ba9-0000-496d-a51e-14679cd37e2b" containerID="17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925" exitCode=0 Jun 08 11:43:35.163841 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.163793 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" Jun 08 11:43:35.163841 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.163828 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" event={"ID":"7b672ba9-0000-496d-a51e-14679cd37e2b","Type":"ContainerDied","Data":"17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925"} Jun 08 11:43:35.164055 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.163867 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff68ccc45-5hc4t" event={"ID":"7b672ba9-0000-496d-a51e-14679cd37e2b","Type":"ContainerDied","Data":"e7fbef2e7e4699f4718f87f9c24e7abf4c7fa90957a855f64711d7c97670e5a2"} Jun 08 11:43:35.164055 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.163899 2562 scope.go:117] "RemoveContainer" containerID="17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925" Jun 08 11:43:35.177374 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.177355 2562 scope.go:117] "RemoveContainer" containerID="17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925" Jun 08 11:43:35.177714 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:43:35.177680 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925\": container with ID starting with 17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925 not found: ID does not exist" containerID="17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925" Jun 08 11:43:35.177793 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.177726 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925"} err="failed to get container status \"17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925\": rpc error: code = NotFound desc = could not find container \"17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925\": container with ID starting with 17cbff3b376d16cd7235c07d23824e7ddf5bf541c62594d111dec2a5ce9b4925 not found: ID does not exist" Jun 08 11:43:35.187699 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.187669 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ff68ccc45-5hc4t"] Jun 08 11:43:35.190930 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.190904 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6ff68ccc45-5hc4t"] Jun 08 11:43:35.202439 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:35.202408 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b672ba9-0000-496d-a51e-14679cd37e2b" path="/var/lib/kubelet/pods/7b672ba9-0000-496d-a51e-14679cd37e2b/volumes" Jun 08 11:43:40.126913 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:40.126878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:40.191850 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:40.191797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:42.193258 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:42.193220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:42.773955 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:43:42.773902 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-74ptp" podUID="4cad1fdb-2d67-433d-96b9-b2f815229312" Jun 08 11:43:43.193961 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:43.193924 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74ptp" Jun 08 11:43:44.127387 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:44.127335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:44.183723 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:44.183691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:46.053920 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:46.053870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:43:46.056707 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:46.056678 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cad1fdb-2d67-433d-96b9-b2f815229312-metrics-tls\") pod \"dns-default-74ptp\" (UID: \"4cad1fdb-2d67-433d-96b9-b2f815229312\") " pod="openshift-dns/dns-default-74ptp" Jun 08 11:43:46.197333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:46.197297 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9sblq\"" Jun 08 11:43:46.205604 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:46.205576 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-74ptp" Jun 08 11:43:46.353926 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:46.353843 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-74ptp"] Jun 08 11:43:46.357121 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:43:46.357085 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cad1fdb_2d67_433d_96b9_b2f815229312.slice/crio-a8ddcb39ebd4c501752e0981c38c66c1669b589615927f5c80b2e3433102704e WatchSource:0}: Error finding container a8ddcb39ebd4c501752e0981c38c66c1669b589615927f5c80b2e3433102704e: Status 404 returned error can't find the container with id a8ddcb39ebd4c501752e0981c38c66c1669b589615927f5c80b2e3433102704e Jun 08 11:43:47.211410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:47.211372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-74ptp" event={"ID":"4cad1fdb-2d67-433d-96b9-b2f815229312","Type":"ContainerStarted","Data":"a8ddcb39ebd4c501752e0981c38c66c1669b589615927f5c80b2e3433102704e"} Jun 08 11:43:48.220655 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:48.220571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-74ptp" event={"ID":"4cad1fdb-2d67-433d-96b9-b2f815229312","Type":"ContainerStarted","Data":"5a8bec14824d8eb2cba917eb184e39f213e2ea76ea9b3adf0e82f370c3aa3b95"} Jun 08 11:43:49.225944 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:49.225908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-74ptp" event={"ID":"4cad1fdb-2d67-433d-96b9-b2f815229312","Type":"ContainerStarted","Data":"3de2288722a2bd2de7620742f15fc40507680a6ea9b6484f630051682729c4cc"} Jun 08 11:43:49.226374 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:49.226008 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-74ptp" Jun 08 11:43:49.243092 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:49.243035 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-74ptp" podStartSLOduration=251.504159338 podStartE2EDuration="4m13.243018547s" totalImagesPullingTime="1.738859209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:43:49.241242316 +0000 UTC m=+288.543019013" watchObservedRunningTime="2026-06-08 11:43:49.243018547 +0000 UTC m=+288.544795244" Jun 08 11:43:49.828171 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:49.828142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:50.942598 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:50.942549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:51.972364 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:51.972315 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6fd4cb8d68-cp9cw" podUID="fb5f7442-3594-46d2-9dc2-9928cf6016eb" containerName="console" containerID="cri-o://eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37" gracePeriod=15 Jun 08 11:43:52.217513 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.217481 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:52.241963 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.241857 2562 generic.go:356] "Generic (PLEG): container finished" podID="fb5f7442-3594-46d2-9dc2-9928cf6016eb" containerID="eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37" exitCode=0 Jun 08 11:43:52.241963 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.241925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd4cb8d68-cp9cw" event={"ID":"fb5f7442-3594-46d2-9dc2-9928cf6016eb","Type":"ContainerDied","Data":"eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37"} Jun 08 11:43:52.241963 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.241937 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd4cb8d68-cp9cw" Jun 08 11:43:52.242226 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.241998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd4cb8d68-cp9cw" event={"ID":"fb5f7442-3594-46d2-9dc2-9928cf6016eb","Type":"ContainerDied","Data":"c529680fccfb674193c7305e1a48cdb5ced6319250d2626271df635592629090"} Jun 08 11:43:52.242226 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.242027 2562 scope.go:117] "RemoveContainer" containerID="eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37" Jun 08 11:43:52.251219 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.251194 2562 scope.go:117] "RemoveContainer" containerID="eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37" Jun 08 11:43:52.251602 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:43:52.251581 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37\": container with ID starting with eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37 not found: ID does not exist" containerID="eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37" Jun 08 11:43:52.251696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.251636 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37"} err="failed to get container status \"eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37\": rpc error: code = NotFound desc = could not find container \"eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37\": container with ID starting with eee517f4637703558d3dbca3fd85208f430c99f99cd054d67f84d2e2e205af37 not found: ID does not exist" Jun 08 11:43:52.404203 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404145 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-oauth-serving-cert\") pod \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " Jun 08 11:43:52.404203 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404206 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-992c9\" (UniqueName: \"kubernetes.io/projected/fb5f7442-3594-46d2-9dc2-9928cf6016eb-kube-api-access-992c9\") pod \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " Jun 08 11:43:52.404469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404307 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-config\") pod \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " Jun 08 11:43:52.404469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404347 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-serving-cert\") pod \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " Jun 08 11:43:52.404469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404372 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-oauth-config\") pod \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " Jun 08 11:43:52.404469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404413 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-service-ca\") pod \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\" (UID: \"fb5f7442-3594-46d2-9dc2-9928cf6016eb\") " Jun 08 11:43:52.404709 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404602 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb5f7442-3594-46d2-9dc2-9928cf6016eb" (UID: "fb5f7442-3594-46d2-9dc2-9928cf6016eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:43:52.404838 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404808 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-config" (OuterVolumeSpecName: "console-config") pod "fb5f7442-3594-46d2-9dc2-9928cf6016eb" (UID: "fb5f7442-3594-46d2-9dc2-9928cf6016eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:43:52.404966 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.404871 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb5f7442-3594-46d2-9dc2-9928cf6016eb" (UID: "fb5f7442-3594-46d2-9dc2-9928cf6016eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:43:52.406850 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.406809 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb5f7442-3594-46d2-9dc2-9928cf6016eb" (UID: "fb5f7442-3594-46d2-9dc2-9928cf6016eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:43:52.406850 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.406831 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb5f7442-3594-46d2-9dc2-9928cf6016eb" (UID: "fb5f7442-3594-46d2-9dc2-9928cf6016eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:43:52.407031 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.406860 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5f7442-3594-46d2-9dc2-9928cf6016eb-kube-api-access-992c9" (OuterVolumeSpecName: "kube-api-access-992c9") pod "fb5f7442-3594-46d2-9dc2-9928cf6016eb" (UID: "fb5f7442-3594-46d2-9dc2-9928cf6016eb"). InnerVolumeSpecName "kube-api-access-992c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:43:52.505886 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.505782 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:52.505886 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.505827 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:52.505886 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.505839 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb5f7442-3594-46d2-9dc2-9928cf6016eb-console-oauth-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:52.505886 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.505848 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-service-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:52.505886 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.505857 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb5f7442-3594-46d2-9dc2-9928cf6016eb-oauth-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:52.505886 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.505865 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-992c9\" (UniqueName: \"kubernetes.io/projected/fb5f7442-3594-46d2-9dc2-9928cf6016eb-kube-api-access-992c9\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:43:52.562783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.562739 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd4cb8d68-cp9cw"] Jun 08 11:43:52.568649 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:52.568599 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fd4cb8d68-cp9cw"] Jun 08 11:43:53.202757 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:53.202720 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5f7442-3594-46d2-9dc2-9928cf6016eb" path="/var/lib/kubelet/pods/fb5f7442-3594-46d2-9dc2-9928cf6016eb/volumes" Jun 08 11:43:55.446042 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:55.446004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:55.594804 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:55.594770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:55.687546 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:55.687508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:55.741273 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:55.741184 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:56.199584 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:56.199530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:56.253663 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:56.253601 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:57.458974 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:57.458939 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:57.514831 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:57.514799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:57.573180 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:57.573143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:43:59.233449 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:59.233421 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-74ptp" Jun 08 11:43:59.983827 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:43:59.983794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:00.046454 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:00.046427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:01.145892 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:01.145863 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:44:01.147981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:01.147870 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:44:01.151421 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:01.151396 2562 kubelet.go:1648] "Image garbage collection succeeded" Jun 08 11:44:01.822538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:01.822505 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:01.878726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:01.878693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:04.468884 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:04.468847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:04.526421 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:04.526396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:08.356564 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:08.356530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:10.895272 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:10.895236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:10.996312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:10.996279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:14.324072 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.323977 2562 generic.go:356] "Generic (PLEG): container finished" podID="aecca07d-03c6-4998-947c-a42108f7ac51" containerID="4d4206684be7a454a01a78b8e20adc39cc72d182313abb98de8f6e83a41d1604" exitCode=0 Jun 08 11:44:14.324499 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.324067 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" event={"ID":"aecca07d-03c6-4998-947c-a42108f7ac51","Type":"ContainerDied","Data":"4d4206684be7a454a01a78b8e20adc39cc72d182313abb98de8f6e83a41d1604"} Jun 08 11:44:14.324499 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.324410 2562 scope.go:117] "RemoveContainer" containerID="4d4206684be7a454a01a78b8e20adc39cc72d182313abb98de8f6e83a41d1604" Jun 08 11:44:14.325591 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.325565 2562 generic.go:356] "Generic (PLEG): container finished" podID="a3a0e723-0c3c-4385-a198-ee3b4746fb94" containerID="bca49ef23c32e43a6e9a5ef11a9d91cd80f2caffc230db88213ef7c2631028ac" exitCode=0 Jun 08 11:44:14.325718 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.325637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-4bxmm" event={"ID":"a3a0e723-0c3c-4385-a198-ee3b4746fb94","Type":"ContainerDied","Data":"bca49ef23c32e43a6e9a5ef11a9d91cd80f2caffc230db88213ef7c2631028ac"} Jun 08 11:44:14.325931 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.325914 2562 scope.go:117] "RemoveContainer" containerID="bca49ef23c32e43a6e9a5ef11a9d91cd80f2caffc230db88213ef7c2631028ac" Jun 08 11:44:14.911285 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.911253 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c75c5578f-n574c_86fee2ac-8922-4a3c-890c-760527f5ea2f/router/0.log" Jun 08 11:44:14.936561 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:14.936529 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7kbj_009f5259-e6a0-45d9-87e3-042f9313cf92/serve-healthcheck-canary/0.log" Jun 08 11:44:15.331947 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:15.331908 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-896549976-4bxmm" event={"ID":"a3a0e723-0c3c-4385-a198-ee3b4746fb94","Type":"ContainerStarted","Data":"3bf7a1687eb10842dc980f279c24304b7610e11df4ab2c06bd95f1ee114519a9"} Jun 08 11:44:15.333763 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:15.333735 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-6bc5ff68f5-lhmlx" event={"ID":"aecca07d-03c6-4998-947c-a42108f7ac51","Type":"ContainerStarted","Data":"9539d0803cca7e4eeec822b96450b595b69a7dfc536d6fc061a8a157565d1794"} Jun 08 11:44:16.334964 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:16.334902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:16.434403 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:16.434372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:23.444833 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:23.444799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:23.503598 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:23.503567 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:30.109788 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:30.109748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:30.208955 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:30.208916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:30.267656 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:30.267586 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:37.311370 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:37.311334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:37.365527 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:37.365494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:38.447441 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:38.447409 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:38.512772 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:38.512742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:41.349244 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:41.349218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:41.745786 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:41.745713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:42.422467 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:42.422436 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:44:42.422844 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:42.422718 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="alertmanager" containerID="cri-o://51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0" gracePeriod=120 Jun 08 11:44:42.422844 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:42.422801 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-metric" containerID="cri-o://54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26" gracePeriod=120 Jun 08 11:44:42.422962 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:42.422834 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="prom-label-proxy" containerID="cri-o://da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729" gracePeriod=120 Jun 08 11:44:42.422962 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:42.422854 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-web" containerID="cri-o://69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6" gracePeriod=120 Jun 08 11:44:42.422962 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:42.422857 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy" containerID="cri-o://ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8" gracePeriod=120 Jun 08 11:44:42.422962 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:42.422865 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="config-reloader" containerID="cri-o://cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103" gracePeriod=120 Jun 08 11:44:43.440915 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.440884 2562 generic.go:356] "Generic (PLEG): container finished" podID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerID="da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729" exitCode=0 Jun 08 11:44:43.440915 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.440908 2562 generic.go:356] "Generic (PLEG): container finished" podID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerID="ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8" exitCode=0 Jun 08 11:44:43.440915 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.440915 2562 generic.go:356] "Generic (PLEG): container finished" podID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerID="cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103" exitCode=0 Jun 08 11:44:43.440915 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.440920 2562 generic.go:356] "Generic (PLEG): container finished" podID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerID="51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0" exitCode=0 Jun 08 11:44:43.441345 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.440953 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729"} Jun 08 11:44:43.441345 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.441011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8"} Jun 08 11:44:43.441345 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.441039 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103"} Jun 08 11:44:43.441345 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.441062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0"} Jun 08 11:44:43.667361 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.667340 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:43.797223 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797198 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-main-db\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797380 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797230 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-out\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797380 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797257 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-web\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797380 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797308 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-volume\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797380 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797357 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-main-tls\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797537 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797404 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797537 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797431 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797537 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797458 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-metrics-client-ca\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797537 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797486 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-web-config\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797537 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797519 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-tls-assets\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797546 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvgcj\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-kube-api-access-vvgcj\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797566 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:44:43.797800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797581 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-cluster-tls-config\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.797800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797606 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"43b576bc-41ae-49de-8b23-ff6f5e262cac\" (UID: \"43b576bc-41ae-49de-8b23-ff6f5e262cac\") " Jun 08 11:44:43.798005 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797810 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:44:43.798005 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.797862 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-main-db\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.800112 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.800084 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:43.800247 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.800128 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-volume" (OuterVolumeSpecName: "config-volume") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:43.800247 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.800155 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-out" (OuterVolumeSpecName: "config-out") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:44:43.800454 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.800420 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-kube-api-access-vvgcj" (OuterVolumeSpecName: "kube-api-access-vvgcj") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "kube-api-access-vvgcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:44:43.800550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.800490 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:44:43.800877 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.800850 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:43.801070 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.801046 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:44:43.801999 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.801974 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:43.802141 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.802125 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:43.805329 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.805308 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:43.811420 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.811400 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-web-config" (OuterVolumeSpecName: "web-config") pod "43b576bc-41ae-49de-8b23-ff6f5e262cac" (UID: "43b576bc-41ae-49de-8b23-ff6f5e262cac"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:44:43.898894 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898862 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-metrics-client-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.898894 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898892 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-web-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.898894 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898902 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-tls-assets\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898910 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgcj\" (UniqueName: \"kubernetes.io/projected/43b576bc-41ae-49de-8b23-ff6f5e262cac-kube-api-access-vvgcj\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898918 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-cluster-tls-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898928 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898936 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-out\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898944 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898952 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-config-volume\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898960 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-main-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898969 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43b576bc-41ae-49de-8b23-ff6f5e262cac-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:43.899097 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:43.898978 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43b576bc-41ae-49de-8b23-ff6f5e262cac-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:44:44.447829 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.447802 2562 generic.go:356] "Generic (PLEG): container finished" podID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerID="54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26" exitCode=0 Jun 08 11:44:44.447829 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.447825 2562 generic.go:356] "Generic (PLEG): container finished" podID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerID="69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6" exitCode=0 Jun 08 11:44:44.448215 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.447878 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.448215 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.447876 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26"} Jun 08 11:44:44.448215 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.448000 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6"} Jun 08 11:44:44.448215 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.448029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43b576bc-41ae-49de-8b23-ff6f5e262cac","Type":"ContainerDied","Data":"0f355d890df82ad011685cb5173f60d673a3cd04648a91cdd54c36fff4c6c1a2"} Jun 08 11:44:44.448215 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.448057 2562 scope.go:117] "RemoveContainer" containerID="da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729" Jun 08 11:44:44.455801 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.455785 2562 scope.go:117] "RemoveContainer" containerID="54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26" Jun 08 11:44:44.462281 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.462266 2562 scope.go:117] "RemoveContainer" containerID="ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8" Jun 08 11:44:44.468465 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.468449 2562 scope.go:117] "RemoveContainer" containerID="69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6" Jun 08 11:44:44.470921 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.470901 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:44:44.475074 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.475055 2562 scope.go:117] "RemoveContainer" containerID="cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103" Jun 08 11:44:44.476941 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.476924 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:44:44.481808 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.481793 2562 scope.go:117] "RemoveContainer" containerID="51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0" Jun 08 11:44:44.488146 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.488130 2562 scope.go:117] "RemoveContainer" containerID="c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510" Jun 08 11:44:44.496173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.496144 2562 scope.go:117] "RemoveContainer" containerID="da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729" Jun 08 11:44:44.496669 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:44:44.496391 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729\": container with ID starting with da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729 not found: ID does not exist" containerID="da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729" Jun 08 11:44:44.496669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.496423 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729"} err="failed to get container status \"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729\": rpc error: code = NotFound desc = could not find container \"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729\": container with ID starting with da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729 not found: ID does not exist" Jun 08 11:44:44.496669 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.496440 2562 scope.go:117] "RemoveContainer" containerID="54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26" Jun 08 11:44:44.496919 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:44:44.496676 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26\": container with ID starting with 54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26 not found: ID does not exist" containerID="54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26" Jun 08 11:44:44.496919 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.496697 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26"} err="failed to get container status \"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26\": rpc error: code = NotFound desc = could not find container \"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26\": container with ID starting with 54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26 not found: ID does not exist" Jun 08 11:44:44.496919 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.496713 2562 scope.go:117] "RemoveContainer" containerID="ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8" Jun 08 11:44:44.497143 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:44:44.496934 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8\": container with ID starting with ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8 not found: ID does not exist" containerID="ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8" Jun 08 11:44:44.497143 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.496966 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8"} err="failed to get container status \"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8\": rpc error: code = NotFound desc = could not find container \"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8\": container with ID starting with ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8 not found: ID does not exist" Jun 08 11:44:44.497143 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.496989 2562 scope.go:117] "RemoveContainer" containerID="69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6" Jun 08 11:44:44.497359 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:44:44.497243 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6\": container with ID starting with 69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6 not found: ID does not exist" containerID="69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6" Jun 08 11:44:44.497359 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.497269 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6"} err="failed to get container status \"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6\": rpc error: code = NotFound desc = could not find container \"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6\": container with ID starting with 69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6 not found: ID does not exist" Jun 08 11:44:44.497359 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.497287 2562 scope.go:117] "RemoveContainer" containerID="cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103" Jun 08 11:44:44.497677 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:44:44.497545 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103\": container with ID starting with cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103 not found: ID does not exist" containerID="cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103" Jun 08 11:44:44.497677 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.497572 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103"} err="failed to get container status \"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103\": rpc error: code = NotFound desc = could not find container \"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103\": container with ID starting with cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103 not found: ID does not exist" Jun 08 11:44:44.497677 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.497593 2562 scope.go:117] "RemoveContainer" containerID="51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0" Jun 08 11:44:44.497904 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:44:44.497872 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0\": container with ID starting with 51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0 not found: ID does not exist" containerID="51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0" Jun 08 11:44:44.497964 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.497899 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0"} err="failed to get container status \"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0\": rpc error: code = NotFound desc = could not find container \"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0\": container with ID starting with 51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0 not found: ID does not exist" Jun 08 11:44:44.497964 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.497919 2562 scope.go:117] "RemoveContainer" containerID="c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510" Jun 08 11:44:44.498196 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:44:44.498172 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510\": container with ID starting with c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510 not found: ID does not exist" containerID="c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510" Jun 08 11:44:44.498304 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.498202 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510"} err="failed to get container status \"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510\": rpc error: code = NotFound desc = could not find container \"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510\": container with ID starting with c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510 not found: ID does not exist" Jun 08 11:44:44.498304 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.498221 2562 scope.go:117] "RemoveContainer" containerID="da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729" Jun 08 11:44:44.498483 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.498450 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729"} err="failed to get container status \"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729\": rpc error: code = NotFound desc = could not find container \"da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729\": container with ID starting with da375ebefd55efd5e76610083f54babd8adfc88cdadd57fc7cf30b83d8b1b729 not found: ID does not exist" Jun 08 11:44:44.498483 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.498476 2562 scope.go:117] "RemoveContainer" containerID="54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26" Jun 08 11:44:44.498754 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.498722 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26"} err="failed to get container status \"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26\": rpc error: code = NotFound desc = could not find container \"54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26\": container with ID starting with 54ec9fa3dab0c1410510f8f0e652c36280e570fe2776c65bba9bfe9eca114b26 not found: ID does not exist" Jun 08 11:44:44.498754 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.498749 2562 scope.go:117] "RemoveContainer" containerID="ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8" Jun 08 11:44:44.499026 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499006 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8"} err="failed to get container status \"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8\": rpc error: code = NotFound desc = could not find container \"ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8\": container with ID starting with ba457e1fa1ccd9fcc5b6004056b61d153402130aa8e7cb98617d28537bc9dcd8 not found: ID does not exist" Jun 08 11:44:44.499026 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499026 2562 scope.go:117] "RemoveContainer" containerID="69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6" Jun 08 11:44:44.499254 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499228 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6"} err="failed to get container status \"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6\": rpc error: code = NotFound desc = could not find container \"69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6\": container with ID starting with 69deb0744321fb0729025deb013af3d74e6a2400cd1dc123f10de7fa0feaa3a6 not found: ID does not exist" Jun 08 11:44:44.499340 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499254 2562 scope.go:117] "RemoveContainer" containerID="cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103" Jun 08 11:44:44.499516 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499497 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103"} err="failed to get container status \"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103\": rpc error: code = NotFound desc = could not find container \"cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103\": container with ID starting with cea6c95d1f6703335f0741ed57bb673d1156434f4438bc02492b92e013a4b103 not found: ID does not exist" Jun 08 11:44:44.499571 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499517 2562 scope.go:117] "RemoveContainer" containerID="51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0" Jun 08 11:44:44.499731 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499713 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0"} err="failed to get container status \"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0\": rpc error: code = NotFound desc = could not find container \"51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0\": container with ID starting with 51cc2084e439e4ff59c13aac68c39b4c2b628a8259897131cecb23a19ce817f0 not found: ID does not exist" Jun 08 11:44:44.499805 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499734 2562 scope.go:117] "RemoveContainer" containerID="c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510" Jun 08 11:44:44.499964 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.499949 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510"} err="failed to get container status \"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510\": rpc error: code = NotFound desc = could not find container \"c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510\": container with ID starting with c9e83ad73e39f808bab73165b3806294ea8a9ebf1459e5a001b135582405e510 not found: ID does not exist" Jun 08 11:44:44.506173 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506154 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:44:44.506383 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506372 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b672ba9-0000-496d-a51e-14679cd37e2b" containerName="registry" Jun 08 11:44:44.506422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506384 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b672ba9-0000-496d-a51e-14679cd37e2b" containerName="registry" Jun 08 11:44:44.506422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506399 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="prom-label-proxy" Jun 08 11:44:44.506422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506404 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="prom-label-proxy" Jun 08 11:44:44.506422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506414 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f7442-3594-46d2-9dc2-9928cf6016eb" containerName="console" Jun 08 11:44:44.506422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506419 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f7442-3594-46d2-9dc2-9928cf6016eb" containerName="console" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506424 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506429 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506434 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-metric" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506439 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-metric" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506446 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="config-reloader" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506451 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="config-reloader" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506458 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="alertmanager" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506462 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="alertmanager" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506471 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="init-config-reloader" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506476 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="init-config-reloader" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506481 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-web" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506486 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-web" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506524 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-metric" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506531 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f7442-3594-46d2-9dc2-9928cf6016eb" containerName="console" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506536 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506542 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="kube-rbac-proxy-web" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506546 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="prom-label-proxy" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506552 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="alertmanager" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506556 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" containerName="config-reloader" Jun 08 11:44:44.506559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.506562 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b672ba9-0000-496d-a51e-14679cd37e2b" containerName="registry" Jun 08 11:44:44.511172 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.511157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.513571 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.513549 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 08 11:44:44.513673 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.513583 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 08 11:44:44.514250 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.514236 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 08 11:44:44.514336 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.514320 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z2g5g\"" Jun 08 11:44:44.514396 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.514341 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 08 11:44:44.514513 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.514501 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 08 11:44:44.514888 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.514872 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 08 11:44:44.514953 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.514908 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 08 11:44:44.515254 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.515239 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 08 11:44:44.524637 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.524598 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 08 11:44:44.532072 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.532052 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:44:44.604512 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604602 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604516 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604602 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-web-config\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604602 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604716 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604716 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604716 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604716 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-config-out\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604829 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-config-volume\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604829 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604829 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604829 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604821 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.604942 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.604837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhg46\" (UniqueName: \"kubernetes.io/projected/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-kube-api-access-jhg46\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705283 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhg46\" (UniqueName: \"kubernetes.io/projected/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-kube-api-access-jhg46\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-web-config\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705706 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-config-out\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705958 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705751 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-config-volume\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.705958 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.705767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.706913 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.706868 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.707106 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.707086 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.708712 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.708684 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-config-out\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.709057 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.708868 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.709331 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.709107 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.709331 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.709244 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-web-config\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.709830 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.709800 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.709930 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.709844 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.710043 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.710024 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.710149 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.710135 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-config-volume\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.710463 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.710444 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.713026 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.713004 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhg46\" (UniqueName: \"kubernetes.io/projected/400f1501-ccfc-4ddb-b98c-7a243ba7e0ad-kube-api-access-jhg46\") pod \"alertmanager-main-0\" (UID: \"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad\") " pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.820430 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.820409 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 08 11:44:44.952550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:44.952521 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 08 11:44:44.955270 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:44:44.955243 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400f1501_ccfc_4ddb_b98c_7a243ba7e0ad.slice/crio-c53d452c258006cfe9d5a8ec5730799fbd915a8000401ec136dab1f92e1f5a35 WatchSource:0}: Error finding container c53d452c258006cfe9d5a8ec5730799fbd915a8000401ec136dab1f92e1f5a35: Status 404 returned error can't find the container with id c53d452c258006cfe9d5a8ec5730799fbd915a8000401ec136dab1f92e1f5a35 Jun 08 11:44:45.201697 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:45.201666 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43b576bc-41ae-49de-8b23-ff6f5e262cac" path="/var/lib/kubelet/pods/43b576bc-41ae-49de-8b23-ff6f5e262cac/volumes" Jun 08 11:44:45.452796 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:45.452758 2562 generic.go:356] "Generic (PLEG): container finished" podID="400f1501-ccfc-4ddb-b98c-7a243ba7e0ad" containerID="55115e1de1f25a3e245c85f92b8786934efe6454c4c00cf5aa074552fe8e81fe" exitCode=0 Jun 08 11:44:45.453238 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:45.452843 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerDied","Data":"55115e1de1f25a3e245c85f92b8786934efe6454c4c00cf5aa074552fe8e81fe"} Jun 08 11:44:45.453238 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:45.452933 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerStarted","Data":"c53d452c258006cfe9d5a8ec5730799fbd915a8000401ec136dab1f92e1f5a35"} Jun 08 11:44:46.436785 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.436753 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dd869676c-c59qx"] Jun 08 11:44:46.439855 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.439838 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.442120 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.442096 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r7zb2\"" Jun 08 11:44:46.442120 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.442114 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jun 08 11:44:46.442294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.442097 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jun 08 11:44:46.442294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.442224 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jun 08 11:44:46.442294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.442188 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jun 08 11:44:46.442294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.442096 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jun 08 11:44:46.448688 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.448669 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jun 08 11:44:46.451991 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.451972 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dd869676c-c59qx"] Jun 08 11:44:46.466572 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.466547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerStarted","Data":"089ca1931db67b144c1489d171e51c7f9e5813b338bd559fc03e0d3d18a7a67f"} Jun 08 11:44:46.466967 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.466655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerStarted","Data":"ce65f26385c7bfc59027bde86a5d62b68d7f278b15201b22137e0390741fb10c"} Jun 08 11:44:46.466967 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.466682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerStarted","Data":"c8c8f50fc642f99ad0ad1796843b908e4b67b2928c9ade2826eebf2929b568a6"} Jun 08 11:44:46.466967 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.466719 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerStarted","Data":"ccdd3f4f164d787db9e82f15baac2835f6ad6fc0c60404da1e6563e72bf4ddcf"} Jun 08 11:44:46.466967 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.466753 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerStarted","Data":"c04aacfbc2e673c237e94593fd567ced0870bc90d672770c988b1da65757183c"} Jun 08 11:44:46.466967 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.466782 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"400f1501-ccfc-4ddb-b98c-7a243ba7e0ad","Type":"ContainerStarted","Data":"89fd96060d23664d05b3839406ddb7f084f56a11b4f2cbbd40be0836337c6401"} Jun 08 11:44:46.501578 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.501534 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.487742673 podStartE2EDuration="2.501522278s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.779605ms" isStatefulPod=false podCreationTimestamp="2026-06-08 11:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:44:46.500191723 +0000 UTC m=+345.801968418" watchObservedRunningTime="2026-06-08 11:44:46.501522278 +0000 UTC m=+345.803298973" Jun 08 11:44:46.521004 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.520980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj8cx\" (UniqueName: \"kubernetes.io/projected/48221b9d-e13e-457c-80d0-0de595ac8293-kube-api-access-fj8cx\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.521116 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.521021 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-serving-certs-ca-bundle\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.521116 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.521052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-secret-telemeter-client\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.521198 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.521127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-metrics-client-ca\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.521198 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.521154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.521198 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.521171 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.521198 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.521192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-federate-client-tls\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.521366 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.521272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-telemeter-client-tls\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.622520 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.622499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj8cx\" (UniqueName: \"kubernetes.io/projected/48221b9d-e13e-457c-80d0-0de595ac8293-kube-api-access-fj8cx\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-serving-certs-ca-bundle\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623247 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-secret-telemeter-client\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-metrics-client-ca\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623409 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623338 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623409 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-federate-client-tls\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623494 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-telemeter-client-tls\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623906 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623787 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-serving-certs-ca-bundle\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.623906 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.623873 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-metrics-client-ca\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.624395 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.624372 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48221b9d-e13e-457c-80d0-0de595ac8293-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.625732 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.625703 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-secret-telemeter-client\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.625732 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.625719 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.625939 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.625921 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-federate-client-tls\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.625939 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.625934 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/48221b9d-e13e-457c-80d0-0de595ac8293-telemeter-client-tls\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.633285 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.633266 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj8cx\" (UniqueName: \"kubernetes.io/projected/48221b9d-e13e-457c-80d0-0de595ac8293-kube-api-access-fj8cx\") pod \"telemeter-client-5dd869676c-c59qx\" (UID: \"48221b9d-e13e-457c-80d0-0de595ac8293\") " pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.748495 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.748436 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" Jun 08 11:44:46.876739 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.876708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dd869676c-c59qx"] Jun 08 11:44:46.879632 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:44:46.879588 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48221b9d_e13e_457c_80d0_0de595ac8293.slice/crio-c0d247d6b79d950434c6fe9c0a74cfe77922803ed4917b777ff0aa7a8a711974 WatchSource:0}: Error finding container c0d247d6b79d950434c6fe9c0a74cfe77922803ed4917b777ff0aa7a8a711974: Status 404 returned error can't find the container with id c0d247d6b79d950434c6fe9c0a74cfe77922803ed4917b777ff0aa7a8a711974 Jun 08 11:44:46.881535 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:46.881520 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:44:47.472794 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:47.472733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" event={"ID":"48221b9d-e13e-457c-80d0-0de595ac8293","Type":"ContainerStarted","Data":"c0d247d6b79d950434c6fe9c0a74cfe77922803ed4917b777ff0aa7a8a711974"} Jun 08 11:44:49.482294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:49.482260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" event={"ID":"48221b9d-e13e-457c-80d0-0de595ac8293","Type":"ContainerStarted","Data":"f4e2e34c1a5f8261a012b3451533d5ea2e4aa657888c1077c6199be648b22a40"} Jun 08 11:44:49.482689 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:49.482317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" event={"ID":"48221b9d-e13e-457c-80d0-0de595ac8293","Type":"ContainerStarted","Data":"282aae1501d7b6a5b78d0f2c645aacb945bf5e717cc2345b0aa9ff287b0ef4e0"} Jun 08 11:44:49.482689 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:49.482344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" event={"ID":"48221b9d-e13e-457c-80d0-0de595ac8293","Type":"ContainerStarted","Data":"46d9393985449941069f10086ad3842b222b26e9aebdb940386ed4f6189a09be"} Jun 08 11:44:49.503115 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:49.503070 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dd869676c-c59qx" podStartSLOduration=1.771790598 podStartE2EDuration="3.503055801s" totalImagesPullingTime="1.731265203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:44:49.501186123 +0000 UTC m=+348.802962830" watchObservedRunningTime="2026-06-08 11:44:49.503055801 +0000 UTC m=+348.804832521" Jun 08 11:44:50.089675 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.089630 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64dc48759f-5nxbx"] Jun 08 11:44:50.093108 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.093088 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.096503 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.096479 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jun 08 11:44:50.096744 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.096731 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jun 08 11:44:50.097409 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.097390 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rtp72\"" Jun 08 11:44:50.097562 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.097535 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jun 08 11:44:50.097690 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.097566 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jun 08 11:44:50.097690 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.097420 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jun 08 11:44:50.097690 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.097399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jun 08 11:44:50.097690 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.097402 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jun 08 11:44:50.103647 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.103629 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jun 08 11:44:50.103960 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.103939 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64dc48759f-5nxbx"] Jun 08 11:44:50.152749 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.152715 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:50.214783 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.214762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:50.252453 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.252430 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-oauth-config\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.252528 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.252461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-console-config\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.252528 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.252479 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgx9b\" (UniqueName: \"kubernetes.io/projected/eca7a345-138d-4a78-ab70-7f3627986e1c-kube-api-access-dgx9b\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.252634 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.252532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-trusted-ca-bundle\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.252634 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.252565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-serving-cert\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.252634 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.252591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-oauth-serving-cert\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.252734 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.252641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-service-ca\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.291653 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.291587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:50.346906 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.346852 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:50.353569 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.353545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-oauth-config\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.353663 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.353589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-console-config\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.353663 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.353632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgx9b\" (UniqueName: \"kubernetes.io/projected/eca7a345-138d-4a78-ab70-7f3627986e1c-kube-api-access-dgx9b\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.353764 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.353687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-trusted-ca-bundle\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.353764 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.353722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-serving-cert\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.353865 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.353774 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-oauth-serving-cert\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.353865 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.353823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-service-ca\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.354504 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.354485 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-console-config\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.354681 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.354660 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-oauth-serving-cert\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.354734 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.354717 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-trusted-ca-bundle\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.354871 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.354848 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-service-ca\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.356069 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.356040 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-oauth-config\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.356216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.356198 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-serving-cert\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.364333 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.364310 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgx9b\" (UniqueName: \"kubernetes.io/projected/eca7a345-138d-4a78-ab70-7f3627986e1c-kube-api-access-dgx9b\") pod \"console-64dc48759f-5nxbx\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.402406 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.402382 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:44:50.525592 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.525563 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64dc48759f-5nxbx"] Jun 08 11:44:50.529646 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:44:50.529594 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca7a345_138d_4a78_ab70_7f3627986e1c.slice/crio-cf70b6b66a5a46b7e516e883bd014d18d77a6b14682033aa21daaed150727f2f WatchSource:0}: Error finding container cf70b6b66a5a46b7e516e883bd014d18d77a6b14682033aa21daaed150727f2f: Status 404 returned error can't find the container with id cf70b6b66a5a46b7e516e883bd014d18d77a6b14682033aa21daaed150727f2f Jun 08 11:44:50.887817 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.887785 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:50.944084 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:50.944057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:51.491776 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:51.491745 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64dc48759f-5nxbx" event={"ID":"eca7a345-138d-4a78-ab70-7f3627986e1c","Type":"ContainerStarted","Data":"d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57"} Jun 08 11:44:51.491914 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:51.491785 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64dc48759f-5nxbx" event={"ID":"eca7a345-138d-4a78-ab70-7f3627986e1c","Type":"ContainerStarted","Data":"cf70b6b66a5a46b7e516e883bd014d18d77a6b14682033aa21daaed150727f2f"} Jun 08 11:44:51.513038 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:51.512997 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64dc48759f-5nxbx" podStartSLOduration=1.5129806289999999 podStartE2EDuration="1.512980629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:44:51.511509169 +0000 UTC m=+350.813285864" watchObservedRunningTime="2026-06-08 11:44:51.512980629 +0000 UTC m=+350.814757326" Jun 08 11:44:53.445938 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:53.445909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:44:53.504603 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:44:53.504571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:45:00.402968 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:00.402919 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:45:00.403676 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:00.403021 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:45:00.407652 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:00.407631 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:45:00.530247 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:00.530219 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:45:00.730206 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:00.730132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:45:00.879502 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:00.879465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:45:18.826096 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:18.826061 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:45:45.209564 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:45.209532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:45:45.310898 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:45.310863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:45:56.593019 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:56.592981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:45:56.645131 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:45:56.645097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:00.259425 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.259392 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76d9444964-vr488"] Jun 08 11:46:00.262678 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.262654 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.274199 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.274164 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d9444964-vr488"] Jun 08 11:46:00.325049 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.325010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:00.381069 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.381026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-oauth-config\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.381254 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.381080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-service-ca\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.381254 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.381144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-console-config\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.381254 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.381170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-trusted-ca-bundle\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.381254 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.381202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-oauth-serving-cert\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.381414 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.381259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdnk\" (UniqueName: \"kubernetes.io/projected/fab729a7-2a30-4142-995d-caf28bdd3d02-kube-api-access-jzdnk\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.381414 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.381285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-serving-cert\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.391839 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.391807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:00.482720 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.482685 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-console-config\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.482897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.482745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-trusted-ca-bundle\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.482897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.482859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-oauth-serving-cert\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483013 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.482913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdnk\" (UniqueName: \"kubernetes.io/projected/fab729a7-2a30-4142-995d-caf28bdd3d02-kube-api-access-jzdnk\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483013 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.482939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-serving-cert\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483013 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.482985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-oauth-config\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483170 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.483110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-service-ca\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483496 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.483471 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-console-config\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483592 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.483510 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-oauth-serving-cert\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483592 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.483576 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-trusted-ca-bundle\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.483766 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.483748 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-service-ca\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.485527 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.485505 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-serving-cert\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.485722 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.485705 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-oauth-config\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.491568 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.491546 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdnk\" (UniqueName: \"kubernetes.io/projected/fab729a7-2a30-4142-995d-caf28bdd3d02-kube-api-access-jzdnk\") pod \"console-76d9444964-vr488\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.572890 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.572849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:00.734512 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:00.734420 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d9444964-vr488"] Jun 08 11:46:00.737653 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:46:00.737581 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab729a7_2a30_4142_995d_caf28bdd3d02.slice/crio-f326bf3dbd27859524ebd81202fbfcf5b7700abf08d967873c1706e3142b846e WatchSource:0}: Error finding container f326bf3dbd27859524ebd81202fbfcf5b7700abf08d967873c1706e3142b846e: Status 404 returned error can't find the container with id f326bf3dbd27859524ebd81202fbfcf5b7700abf08d967873c1706e3142b846e Jun 08 11:46:01.743748 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:01.743713 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d9444964-vr488" event={"ID":"fab729a7-2a30-4142-995d-caf28bdd3d02","Type":"ContainerStarted","Data":"18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856"} Jun 08 11:46:01.744165 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:01.743760 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d9444964-vr488" event={"ID":"fab729a7-2a30-4142-995d-caf28bdd3d02","Type":"ContainerStarted","Data":"f326bf3dbd27859524ebd81202fbfcf5b7700abf08d967873c1706e3142b846e"} Jun 08 11:46:01.763426 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:01.763373 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76d9444964-vr488" podStartSLOduration=1.763356474 podStartE2EDuration="1.763356474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:46:01.762569523 +0000 UTC m=+421.064346220" watchObservedRunningTime="2026-06-08 11:46:01.763356474 +0000 UTC m=+421.065133231" Jun 08 11:46:08.520846 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:08.520792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:08.621624 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:08.621591 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:10.573324 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:10.573286 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:10.573762 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:10.573342 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:10.578110 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:10.578088 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:10.784596 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:10.784564 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:46:10.837701 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:10.837599 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64dc48759f-5nxbx"] Jun 08 11:46:10.946557 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:10.946522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:11.001387 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:11.001350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:11.060626 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:11.060576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:23.459147 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:23.459045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:23.554184 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:23.554148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:23.719754 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:23.719658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:46:35.861603 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:35.861543 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-64dc48759f-5nxbx" podUID="eca7a345-138d-4a78-ab70-7f3627986e1c" containerName="console" containerID="cri-o://d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57" gracePeriod=15 Jun 08 11:46:36.103678 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.103652 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:46:36.163372 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163275 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-trusted-ca-bundle\") pod \"eca7a345-138d-4a78-ab70-7f3627986e1c\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " Jun 08 11:46:36.163372 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163323 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-oauth-serving-cert\") pod \"eca7a345-138d-4a78-ab70-7f3627986e1c\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " Jun 08 11:46:36.163372 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163355 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgx9b\" (UniqueName: \"kubernetes.io/projected/eca7a345-138d-4a78-ab70-7f3627986e1c-kube-api-access-dgx9b\") pod \"eca7a345-138d-4a78-ab70-7f3627986e1c\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " Jun 08 11:46:36.163671 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163433 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-oauth-config\") pod \"eca7a345-138d-4a78-ab70-7f3627986e1c\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " Jun 08 11:46:36.163671 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163464 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-serving-cert\") pod \"eca7a345-138d-4a78-ab70-7f3627986e1c\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " Jun 08 11:46:36.163671 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163481 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-service-ca\") pod \"eca7a345-138d-4a78-ab70-7f3627986e1c\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " Jun 08 11:46:36.163671 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163552 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-console-config\") pod \"eca7a345-138d-4a78-ab70-7f3627986e1c\" (UID: \"eca7a345-138d-4a78-ab70-7f3627986e1c\") " Jun 08 11:46:36.163878 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163777 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eca7a345-138d-4a78-ab70-7f3627986e1c" (UID: "eca7a345-138d-4a78-ab70-7f3627986e1c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:46:36.163929 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163886 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eca7a345-138d-4a78-ab70-7f3627986e1c" (UID: "eca7a345-138d-4a78-ab70-7f3627986e1c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:46:36.164015 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.163986 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-service-ca" (OuterVolumeSpecName: "service-ca") pod "eca7a345-138d-4a78-ab70-7f3627986e1c" (UID: "eca7a345-138d-4a78-ab70-7f3627986e1c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:46:36.164071 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.164041 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-console-config" (OuterVolumeSpecName: "console-config") pod "eca7a345-138d-4a78-ab70-7f3627986e1c" (UID: "eca7a345-138d-4a78-ab70-7f3627986e1c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:46:36.165801 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.165775 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eca7a345-138d-4a78-ab70-7f3627986e1c" (UID: "eca7a345-138d-4a78-ab70-7f3627986e1c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:46:36.166090 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.166067 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eca7a345-138d-4a78-ab70-7f3627986e1c" (UID: "eca7a345-138d-4a78-ab70-7f3627986e1c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:46:36.166138 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.166067 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca7a345-138d-4a78-ab70-7f3627986e1c-kube-api-access-dgx9b" (OuterVolumeSpecName: "kube-api-access-dgx9b") pod "eca7a345-138d-4a78-ab70-7f3627986e1c" (UID: "eca7a345-138d-4a78-ab70-7f3627986e1c"). InnerVolumeSpecName "kube-api-access-dgx9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:46:36.264877 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.264845 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-console-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:46:36.264877 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.264871 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-trusted-ca-bundle\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:46:36.265066 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.264887 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-oauth-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:46:36.265066 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.264901 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgx9b\" (UniqueName: \"kubernetes.io/projected/eca7a345-138d-4a78-ab70-7f3627986e1c-kube-api-access-dgx9b\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:46:36.265066 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.264912 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-oauth-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:46:36.265066 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.264923 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eca7a345-138d-4a78-ab70-7f3627986e1c-console-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:46:36.265066 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.264934 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eca7a345-138d-4a78-ab70-7f3627986e1c-service-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:46:36.884687 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.884647 2562 generic.go:356] "Generic (PLEG): container finished" podID="eca7a345-138d-4a78-ab70-7f3627986e1c" containerID="d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57" exitCode=0 Jun 08 11:46:36.885145 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.884695 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64dc48759f-5nxbx" Jun 08 11:46:36.885145 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.884706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64dc48759f-5nxbx" event={"ID":"eca7a345-138d-4a78-ab70-7f3627986e1c","Type":"ContainerDied","Data":"d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57"} Jun 08 11:46:36.885145 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.884754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64dc48759f-5nxbx" event={"ID":"eca7a345-138d-4a78-ab70-7f3627986e1c","Type":"ContainerDied","Data":"cf70b6b66a5a46b7e516e883bd014d18d77a6b14682033aa21daaed150727f2f"} Jun 08 11:46:36.885145 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.884780 2562 scope.go:117] "RemoveContainer" containerID="d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57" Jun 08 11:46:36.893732 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.893712 2562 scope.go:117] "RemoveContainer" containerID="d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57" Jun 08 11:46:36.894054 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:46:36.894032 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57\": container with ID starting with d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57 not found: ID does not exist" containerID="d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57" Jun 08 11:46:36.894100 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.894064 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57"} err="failed to get container status \"d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57\": rpc error: code = NotFound desc = could not find container \"d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57\": container with ID starting with d7efb90649418982c40e2ff6c6ab7e0ae47457e895b8bea98655669ff2d88c57 not found: ID does not exist" Jun 08 11:46:36.905584 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.905552 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64dc48759f-5nxbx"] Jun 08 11:46:36.907690 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:36.907664 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64dc48759f-5nxbx"] Jun 08 11:46:37.202831 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:46:37.202753 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca7a345-138d-4a78-ab70-7f3627986e1c" path="/var/lib/kubelet/pods/eca7a345-138d-4a78-ab70-7f3627986e1c/volumes" Jun 08 11:47:05.719837 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:47:05.719800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:47:05.778058 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:47:05.778025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:47:40.146216 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:47:40.146173 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:09.227510 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:09.227424 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:09.327805 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:09.327764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:16.704757 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:16.704719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:16.810469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:16.810430 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:17.461587 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:17.461550 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:48.906060 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:48.906025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:48.965302 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:48.965268 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:48:59.769407 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:48:59.769373 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:49:00.189129 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.189095 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb"] Jun 08 11:49:00.189416 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.189404 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca7a345-138d-4a78-ab70-7f3627986e1c" containerName="console" Jun 08 11:49:00.189461 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.189418 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca7a345-138d-4a78-ab70-7f3627986e1c" containerName="console" Jun 08 11:49:00.189496 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.189473 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca7a345-138d-4a78-ab70-7f3627986e1c" containerName="console" Jun 08 11:49:00.192800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.192783 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.195280 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.195253 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 08 11:49:00.195425 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.195299 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 08 11:49:00.195425 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.195308 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 11:49:00.195425 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.195309 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 08 11:49:00.195570 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.195520 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jun 08 11:49:00.195660 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.195640 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4p56k\"" Jun 08 11:49:00.195922 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.195906 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 08 11:49:00.196374 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.196360 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 11:49:00.207473 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.207450 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb"] Jun 08 11:49:00.348446 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.348404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5jc\" (UniqueName: \"kubernetes.io/projected/cccfb3f6-0dac-452f-9537-0b3676b0950e-kube-api-access-hn5jc\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.348711 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.348468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-offline-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.348711 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.348498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cccfb3f6-0dac-452f-9537-0b3676b0950e-feast-data\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.348711 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.348557 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-ui-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.348711 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.348669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-registry-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.348924 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.348721 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-online-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.450128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.450042 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-ui-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.450128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.450083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-registry-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.450128 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.450117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-online-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.450373 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.450182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5jc\" (UniqueName: \"kubernetes.io/projected/cccfb3f6-0dac-452f-9537-0b3676b0950e-kube-api-access-hn5jc\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.450373 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.450215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-offline-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.450373 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.450242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cccfb3f6-0dac-452f-9537-0b3676b0950e-feast-data\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.450673 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.450651 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cccfb3f6-0dac-452f-9537-0b3676b0950e-feast-data\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.452782 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.452760 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-registry-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.453022 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.453003 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-offline-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.453088 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.453027 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-ui-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.453088 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.453066 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-online-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.459423 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.459398 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5jc\" (UniqueName: \"kubernetes.io/projected/cccfb3f6-0dac-452f-9537-0b3676b0950e-kube-api-access-hn5jc\") pod \"feast-simple-feast-setup-7fb8f4b78b-hgtvb\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.513412 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.513387 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:00.673538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.673516 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb"] Jun 08 11:49:00.675592 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:49:00.675563 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcccfb3f6_0dac_452f_9537_0b3676b0950e.slice/crio-532fbf0c6ab3cc63ee67377ffe92a4b0926a05d468b6f293d47275da4bac4c11 WatchSource:0}: Error finding container 532fbf0c6ab3cc63ee67377ffe92a4b0926a05d468b6f293d47275da4bac4c11: Status 404 returned error can't find the container with id 532fbf0c6ab3cc63ee67377ffe92a4b0926a05d468b6f293d47275da4bac4c11 Jun 08 11:49:00.962666 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:00.962630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:49:01.175785 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:01.175760 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:49:01.176014 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:01.175996 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:49:01.429687 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:01.429647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerStarted","Data":"532fbf0c6ab3cc63ee67377ffe92a4b0926a05d468b6f293d47275da4bac4c11"} Jun 08 11:49:28.550696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:28.549879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerStarted","Data":"0dd3e690063333de9f56b27459ae0ed81134adf98929b435e92eb48230b20737"} Jun 08 11:49:33.571523 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:33.571488 2562 generic.go:356] "Generic (PLEG): container finished" podID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerID="0dd3e690063333de9f56b27459ae0ed81134adf98929b435e92eb48230b20737" exitCode=0 Jun 08 11:49:33.571920 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:33.571559 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerDied","Data":"0dd3e690063333de9f56b27459ae0ed81134adf98929b435e92eb48230b20737"} Jun 08 11:49:34.577282 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:34.577245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerStarted","Data":"65f1b980a195597f3eb1738a060276d4019983460c0fb6684eb0a3d75815a1eb"} Jun 08 11:49:39.599234 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:39.599197 2562 generic.go:356] "Generic (PLEG): container finished" podID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerID="65f1b980a195597f3eb1738a060276d4019983460c0fb6684eb0a3d75815a1eb" exitCode=0 Jun 08 11:49:39.599641 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:39.599267 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerDied","Data":"65f1b980a195597f3eb1738a060276d4019983460c0fb6684eb0a3d75815a1eb"} Jun 08 11:49:39.608548 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:39.608528 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jun 08 11:49:40.611231 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:40.611185 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerStarted","Data":"fa298bc9b55dc4c0cb5f9ed445b391e1cadab9fe1549594f08f0962022e078f1"} Jun 08 11:49:40.611781 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:40.611256 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerStarted","Data":"8ea1290cbc90fa494c4c440e5f0c0050b5c731172f19daac8cff8c04e801e0e3"} Jun 08 11:49:40.611781 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:40.611278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerStarted","Data":"761c8c0fbedf932480994556136dbcf2d216cc7b3d961d58ce0d7ce33f6457b8"} Jun 08 11:49:40.611781 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:40.611299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerStarted","Data":"356c534bb2dc8d4cac8ed1da9aa84d759ceb1b11633c16aafcadafe02bfe257f"} Jun 08 11:49:40.640981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:40.640900 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podStartSLOduration=12.989846092 podStartE2EDuration="40.640878256s" totalImagesPullingTime="27.651032164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:49:40.639769161 +0000 UTC m=+639.941545860" watchObservedRunningTime="2026-06-08 11:49:40.640878256 +0000 UTC m=+639.942654954" Jun 08 11:49:42.513989 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.513943 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:42.514491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.514016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:42.514491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.514036 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:42.514491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.514053 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:42.516245 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.516203 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jun 08 11:49:42.516491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.516233 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused" Jun 08 11:49:42.516578 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.516252 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jun 08 11:49:42.516578 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:42.516265 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused" Jun 08 11:49:45.514900 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.514870 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.515395 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515020 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.515451 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.515451 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515441 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.515561 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515463 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.515561 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515480 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.515800 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515780 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.515922 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515897 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.516004 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.515953 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.519880 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.519855 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.638063 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.638034 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.640902 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.640879 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:49:45.967456 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:45.967421 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:49:53.803716 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:53.803681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:49:53.860145 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:49:53.860115 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:04.275500 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:04.275459 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d9444964-vr488"] Jun 08 11:50:04.376384 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:04.376351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:04.432337 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:04.432304 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:04.572352 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:04.572317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:04.627767 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:04.627734 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:29.297312 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.297266 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-76d9444964-vr488" podUID="fab729a7-2a30-4142-995d-caf28bdd3d02" containerName="console" containerID="cri-o://18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856" gracePeriod=15 Jun 08 11:50:29.542448 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.542423 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:50:29.601723 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.601637 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-service-ca\") pod \"fab729a7-2a30-4142-995d-caf28bdd3d02\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " Jun 08 11:50:29.601723 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.601697 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-trusted-ca-bundle\") pod \"fab729a7-2a30-4142-995d-caf28bdd3d02\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " Jun 08 11:50:29.601723 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.601715 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-console-config\") pod \"fab729a7-2a30-4142-995d-caf28bdd3d02\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " Jun 08 11:50:29.601970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.601823 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-oauth-serving-cert\") pod \"fab729a7-2a30-4142-995d-caf28bdd3d02\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " Jun 08 11:50:29.601970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.601860 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzdnk\" (UniqueName: \"kubernetes.io/projected/fab729a7-2a30-4142-995d-caf28bdd3d02-kube-api-access-jzdnk\") pod \"fab729a7-2a30-4142-995d-caf28bdd3d02\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " Jun 08 11:50:29.601970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.601902 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-oauth-config\") pod \"fab729a7-2a30-4142-995d-caf28bdd3d02\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " Jun 08 11:50:29.601970 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.601922 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-serving-cert\") pod \"fab729a7-2a30-4142-995d-caf28bdd3d02\" (UID: \"fab729a7-2a30-4142-995d-caf28bdd3d02\") " Jun 08 11:50:29.602178 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.602097 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-console-config" (OuterVolumeSpecName: "console-config") pod "fab729a7-2a30-4142-995d-caf28bdd3d02" (UID: "fab729a7-2a30-4142-995d-caf28bdd3d02"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:50:29.602178 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.602135 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fab729a7-2a30-4142-995d-caf28bdd3d02" (UID: "fab729a7-2a30-4142-995d-caf28bdd3d02"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:50:29.602178 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.602155 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-service-ca" (OuterVolumeSpecName: "service-ca") pod "fab729a7-2a30-4142-995d-caf28bdd3d02" (UID: "fab729a7-2a30-4142-995d-caf28bdd3d02"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:50:29.602321 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.602193 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fab729a7-2a30-4142-995d-caf28bdd3d02" (UID: "fab729a7-2a30-4142-995d-caf28bdd3d02"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:50:29.602321 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.602206 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-console-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:50:29.604256 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.604230 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fab729a7-2a30-4142-995d-caf28bdd3d02" (UID: "fab729a7-2a30-4142-995d-caf28bdd3d02"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:50:29.604339 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.604323 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fab729a7-2a30-4142-995d-caf28bdd3d02" (UID: "fab729a7-2a30-4142-995d-caf28bdd3d02"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:50:29.604393 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.604372 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab729a7-2a30-4142-995d-caf28bdd3d02-kube-api-access-jzdnk" (OuterVolumeSpecName: "kube-api-access-jzdnk") pod "fab729a7-2a30-4142-995d-caf28bdd3d02" (UID: "fab729a7-2a30-4142-995d-caf28bdd3d02"). InnerVolumeSpecName "kube-api-access-jzdnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:50:29.702955 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.702914 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-oauth-config\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:50:29.702955 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.702947 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab729a7-2a30-4142-995d-caf28bdd3d02-console-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:50:29.702955 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.702958 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-service-ca\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:50:29.703170 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.702967 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-trusted-ca-bundle\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:50:29.703170 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.702976 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab729a7-2a30-4142-995d-caf28bdd3d02-oauth-serving-cert\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:50:29.703170 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.702984 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdnk\" (UniqueName: \"kubernetes.io/projected/fab729a7-2a30-4142-995d-caf28bdd3d02-kube-api-access-jzdnk\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:50:29.806717 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.806682 2562 generic.go:356] "Generic (PLEG): container finished" podID="fab729a7-2a30-4142-995d-caf28bdd3d02" containerID="18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856" exitCode=0 Jun 08 11:50:29.806897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.806730 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d9444964-vr488" Jun 08 11:50:29.806897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.806751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d9444964-vr488" event={"ID":"fab729a7-2a30-4142-995d-caf28bdd3d02","Type":"ContainerDied","Data":"18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856"} Jun 08 11:50:29.806897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.806794 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d9444964-vr488" event={"ID":"fab729a7-2a30-4142-995d-caf28bdd3d02","Type":"ContainerDied","Data":"f326bf3dbd27859524ebd81202fbfcf5b7700abf08d967873c1706e3142b846e"} Jun 08 11:50:29.806897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.806816 2562 scope.go:117] "RemoveContainer" containerID="18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856" Jun 08 11:50:29.815551 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.815532 2562 scope.go:117] "RemoveContainer" containerID="18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856" Jun 08 11:50:29.815843 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:50:29.815821 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856\": container with ID starting with 18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856 not found: ID does not exist" containerID="18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856" Jun 08 11:50:29.815921 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.815851 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856"} err="failed to get container status \"18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856\": rpc error: code = NotFound desc = could not find container \"18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856\": container with ID starting with 18fe546f165ec17c1df2f4ce04754a87570312b5cab2e3add650a0a10e5a2856 not found: ID does not exist" Jun 08 11:50:29.827105 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.827082 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d9444964-vr488"] Jun 08 11:50:29.831266 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:29.831242 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76d9444964-vr488"] Jun 08 11:50:31.202189 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:31.202157 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab729a7-2a30-4142-995d-caf28bdd3d02" path="/var/lib/kubelet/pods/fab729a7-2a30-4142-995d-caf28bdd3d02/volumes" Jun 08 11:50:47.534719 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.534685 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb"] Jun 08 11:50:47.535217 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.534916 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="registry" containerID="cri-o://356c534bb2dc8d4cac8ed1da9aa84d759ceb1b11633c16aafcadafe02bfe257f" gracePeriod=30 Jun 08 11:50:47.535217 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.534981 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="offline" containerID="cri-o://8ea1290cbc90fa494c4c440e5f0c0050b5c731172f19daac8cff8c04e801e0e3" gracePeriod=30 Jun 08 11:50:47.535217 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.534998 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" containerID="cri-o://761c8c0fbedf932480994556136dbcf2d216cc7b3d961d58ce0d7ce33f6457b8" gracePeriod=30 Jun 08 11:50:47.535217 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.535071 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" containerID="cri-o://fa298bc9b55dc4c0cb5f9ed445b391e1cadab9fe1549594f08f0962022e078f1" gracePeriod=30 Jun 08 11:50:47.536525 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.536499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:47.632307 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.632274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:47.735836 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.735799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:47.793563 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.793468 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr"] Jun 08 11:50:47.793889 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.793870 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab729a7-2a30-4142-995d-caf28bdd3d02" containerName="console" Jun 08 11:50:47.793975 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.793892 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab729a7-2a30-4142-995d-caf28bdd3d02" containerName="console" Jun 08 11:50:47.793975 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.793965 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab729a7-2a30-4142-995d-caf28bdd3d02" containerName="console" Jun 08 11:50:47.798638 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.798590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:47.800905 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.800886 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-smp92\"" Jun 08 11:50:47.811434 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.811409 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr"] Jun 08 11:50:47.816535 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.816509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:47.880737 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.880707 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:47.949364 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.949323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-ui-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:47.949364 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.949366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31458c41-68c6-44fc-bafe-863b3d285b9a-feast-data\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:47.949632 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.949390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-registry-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:47.949632 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.949477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-online-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:47.949632 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.949509 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlm92\" (UniqueName: \"kubernetes.io/projected/31458c41-68c6-44fc-bafe-863b3d285b9a-kube-api-access-mlm92\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:47.949632 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.949530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-offline-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:47.978695 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:47.978665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:48.050308 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.050284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-online-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.050396 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.050319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlm92\" (UniqueName: \"kubernetes.io/projected/31458c41-68c6-44fc-bafe-863b3d285b9a-kube-api-access-mlm92\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.050396 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.050342 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-offline-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.050503 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.050398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-ui-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.050503 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.050415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31458c41-68c6-44fc-bafe-863b3d285b9a-feast-data\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.050503 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.050437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-registry-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.050861 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.050829 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31458c41-68c6-44fc-bafe-863b3d285b9a-feast-data\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.053073 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.053049 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-registry-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.053207 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.053170 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-ui-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.053337 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.053318 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-online-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.053373 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.053340 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-offline-tls\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.057594 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.057573 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlm92\" (UniqueName: \"kubernetes.io/projected/31458c41-68c6-44fc-bafe-863b3d285b9a-kube-api-access-mlm92\") pod \"feast-simple-feast-setup-7fb8f4b78b-l5zhr\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.079522 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.079494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:48.107716 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.107683 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:50:48.131961 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.131923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:50:48.248885 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.248741 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr"] Jun 08 11:50:48.250958 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:50:48.250923 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31458c41_68c6_44fc_bafe_863b3d285b9a.slice/crio-a1eb59d629a9888cb224f70e102a28a50d41a7a101f3d77a673a0f7fd42967b3 WatchSource:0}: Error finding container a1eb59d629a9888cb224f70e102a28a50d41a7a101f3d77a673a0f7fd42967b3: Status 404 returned error can't find the container with id a1eb59d629a9888cb224f70e102a28a50d41a7a101f3d77a673a0f7fd42967b3 Jun 08 11:50:48.880983 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.880949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerStarted","Data":"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3"} Jun 08 11:50:48.881428 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.880997 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerStarted","Data":"a1eb59d629a9888cb224f70e102a28a50d41a7a101f3d77a673a0f7fd42967b3"} Jun 08 11:50:48.883909 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.883886 2562 generic.go:356] "Generic (PLEG): container finished" podID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerID="fa298bc9b55dc4c0cb5f9ed445b391e1cadab9fe1549594f08f0962022e078f1" exitCode=0 Jun 08 11:50:48.883909 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.883907 2562 generic.go:356] "Generic (PLEG): container finished" podID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerID="761c8c0fbedf932480994556136dbcf2d216cc7b3d961d58ce0d7ce33f6457b8" exitCode=0 Jun 08 11:50:48.884052 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.883943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerDied","Data":"fa298bc9b55dc4c0cb5f9ed445b391e1cadab9fe1549594f08f0962022e078f1"} Jun 08 11:50:48.884052 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:48.883967 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerDied","Data":"761c8c0fbedf932480994556136dbcf2d216cc7b3d961d58ce0d7ce33f6457b8"} Jun 08 11:50:52.900437 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:52.900352 2562 generic.go:356] "Generic (PLEG): container finished" podID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerID="f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3" exitCode=0 Jun 08 11:50:52.900437 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:52.900409 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerDied","Data":"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3"} Jun 08 11:50:53.906420 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:53.906374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerStarted","Data":"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9"} Jun 08 11:50:55.515838 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:55.515789 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jun 08 11:50:55.638269 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:55.638221 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jun 08 11:50:58.927529 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:58.927494 2562 generic.go:356] "Generic (PLEG): container finished" podID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerID="c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9" exitCode=0 Jun 08 11:50:58.927985 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:58.927533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerDied","Data":"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9"} Jun 08 11:50:59.938352 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:59.938311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerStarted","Data":"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92"} Jun 08 11:50:59.938883 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:59.938375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerStarted","Data":"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8"} Jun 08 11:50:59.938883 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:59.938402 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerStarted","Data":"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080"} Jun 08 11:50:59.938883 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:59.938420 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerStarted","Data":"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4"} Jun 08 11:50:59.964044 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:50:59.963942 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podStartSLOduration=12.963918522 podStartE2EDuration="12.963918522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:50:59.961953167 +0000 UTC m=+719.263729887" watchObservedRunningTime="2026-06-08 11:50:59.963918522 +0000 UTC m=+719.265695231" Jun 08 11:51:00.108032 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.107995 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:00.108324 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.108289 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:00.108449 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.108330 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:00.108449 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.108349 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:00.110275 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.110213 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Jun 08 11:51:00.110275 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.110239 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Jun 08 11:51:00.110478 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.110274 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jun 08 11:51:00.110478 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:00.110326 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jun 08 11:51:03.109080 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:03.109028 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Jun 08 11:51:03.109655 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:03.109036 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jun 08 11:51:03.109758 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:03.109047 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Jun 08 11:51:03.109758 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:03.109108 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jun 08 11:51:05.515696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:05.515652 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jun 08 11:51:05.638813 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:05.638768 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jun 08 11:51:06.108897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.108868 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.109066 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.108971 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.109205 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.109185 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.109692 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.109563 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.109692 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.109602 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.109692 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.109638 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.109870 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.109735 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.109870 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.109789 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.110024 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.110012 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.115075 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.115058 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.974078 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.974049 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:06.976971 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:06.976947 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:51:07.195211 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:07.195172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:51:15.515797 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:15.515745 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jun 08 11:51:15.516237 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:15.515904 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:51:15.638449 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:15.638404 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jun 08 11:51:15.638653 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:15.638529 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:51:18.021874 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.021785 2562 generic.go:356] "Generic (PLEG): container finished" podID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerID="8ea1290cbc90fa494c4c440e5f0c0050b5c731172f19daac8cff8c04e801e0e3" exitCode=137 Jun 08 11:51:18.021874 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.021811 2562 generic.go:356] "Generic (PLEG): container finished" podID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerID="356c534bb2dc8d4cac8ed1da9aa84d759ceb1b11633c16aafcadafe02bfe257f" exitCode=137 Jun 08 11:51:18.021874 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.021841 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerDied","Data":"8ea1290cbc90fa494c4c440e5f0c0050b5c731172f19daac8cff8c04e801e0e3"} Jun 08 11:51:18.021874 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.021875 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerDied","Data":"356c534bb2dc8d4cac8ed1da9aa84d759ceb1b11633c16aafcadafe02bfe257f"} Jun 08 11:51:18.180720 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.180694 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:51:18.317641 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.317595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-registry-tls\") pod \"cccfb3f6-0dac-452f-9537-0b3676b0950e\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " Jun 08 11:51:18.317801 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.317699 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-online-tls\") pod \"cccfb3f6-0dac-452f-9537-0b3676b0950e\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " Jun 08 11:51:18.317801 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.317722 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-offline-tls\") pod \"cccfb3f6-0dac-452f-9537-0b3676b0950e\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " Jun 08 11:51:18.317801 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.317738 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cccfb3f6-0dac-452f-9537-0b3676b0950e-feast-data\") pod \"cccfb3f6-0dac-452f-9537-0b3676b0950e\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " Jun 08 11:51:18.317801 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.317756 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-ui-tls\") pod \"cccfb3f6-0dac-452f-9537-0b3676b0950e\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " Jun 08 11:51:18.318011 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.317850 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn5jc\" (UniqueName: \"kubernetes.io/projected/cccfb3f6-0dac-452f-9537-0b3676b0950e-kube-api-access-hn5jc\") pod \"cccfb3f6-0dac-452f-9537-0b3676b0950e\" (UID: \"cccfb3f6-0dac-452f-9537-0b3676b0950e\") " Jun 08 11:51:18.318310 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.318276 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccfb3f6-0dac-452f-9537-0b3676b0950e-feast-data" (OuterVolumeSpecName: "feast-data") pod "cccfb3f6-0dac-452f-9537-0b3676b0950e" (UID: "cccfb3f6-0dac-452f-9537-0b3676b0950e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:51:18.320072 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.320035 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cccfb3f6-0dac-452f-9537-0b3676b0950e" (UID: "cccfb3f6-0dac-452f-9537-0b3676b0950e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:51:18.320072 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.320040 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccfb3f6-0dac-452f-9537-0b3676b0950e-kube-api-access-hn5jc" (OuterVolumeSpecName: "kube-api-access-hn5jc") pod "cccfb3f6-0dac-452f-9537-0b3676b0950e" (UID: "cccfb3f6-0dac-452f-9537-0b3676b0950e"). InnerVolumeSpecName "kube-api-access-hn5jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:51:18.320241 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.320104 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-online-tls" (OuterVolumeSpecName: "online-tls") pod "cccfb3f6-0dac-452f-9537-0b3676b0950e" (UID: "cccfb3f6-0dac-452f-9537-0b3676b0950e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:51:18.320386 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.320370 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "cccfb3f6-0dac-452f-9537-0b3676b0950e" (UID: "cccfb3f6-0dac-452f-9537-0b3676b0950e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:51:18.320468 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.320441 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "cccfb3f6-0dac-452f-9537-0b3676b0950e" (UID: "cccfb3f6-0dac-452f-9537-0b3676b0950e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:51:18.418675 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.418646 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5jc\" (UniqueName: \"kubernetes.io/projected/cccfb3f6-0dac-452f-9537-0b3676b0950e-kube-api-access-hn5jc\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:51:18.418675 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.418671 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:51:18.418675 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.418680 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:51:18.418846 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.418687 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:51:18.418846 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.418694 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/cccfb3f6-0dac-452f-9537-0b3676b0950e-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:51:18.418846 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:18.418701 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/cccfb3f6-0dac-452f-9537-0b3676b0950e-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:51:19.029076 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.029052 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" Jun 08 11:51:19.029439 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.029072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb" event={"ID":"cccfb3f6-0dac-452f-9537-0b3676b0950e","Type":"ContainerDied","Data":"532fbf0c6ab3cc63ee67377ffe92a4b0926a05d468b6f293d47275da4bac4c11"} Jun 08 11:51:19.029439 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.029144 2562 scope.go:117] "RemoveContainer" containerID="fa298bc9b55dc4c0cb5f9ed445b391e1cadab9fe1549594f08f0962022e078f1" Jun 08 11:51:19.038130 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.038107 2562 scope.go:117] "RemoveContainer" containerID="8ea1290cbc90fa494c4c440e5f0c0050b5c731172f19daac8cff8c04e801e0e3" Jun 08 11:51:19.047971 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.047947 2562 scope.go:117] "RemoveContainer" containerID="761c8c0fbedf932480994556136dbcf2d216cc7b3d961d58ce0d7ce33f6457b8" Jun 08 11:51:19.051124 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.051104 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb"] Jun 08 11:51:19.054809 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.054788 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-hgtvb"] Jun 08 11:51:19.056951 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.056933 2562 scope.go:117] "RemoveContainer" containerID="356c534bb2dc8d4cac8ed1da9aa84d759ceb1b11633c16aafcadafe02bfe257f" Jun 08 11:51:19.064082 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.064058 2562 scope.go:117] "RemoveContainer" containerID="65f1b980a195597f3eb1738a060276d4019983460c0fb6684eb0a3d75815a1eb" Jun 08 11:51:19.070990 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.070976 2562 scope.go:117] "RemoveContainer" containerID="0dd3e690063333de9f56b27459ae0ed81134adf98929b435e92eb48230b20737" Jun 08 11:51:19.207578 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:51:19.207542 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" path="/var/lib/kubelet/pods/cccfb3f6-0dac-452f-9537-0b3676b0950e/volumes" Jun 08 11:52:09.273417 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.273364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:52:09.370144 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.370090 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:52:09.479138 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.479107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:52:09.514392 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514353 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct"] Jun 08 11:52:09.514699 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514686 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="offline" Jun 08 11:52:09.514750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514700 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="offline" Jun 08 11:52:09.514750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514716 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="registry" Jun 08 11:52:09.514750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514722 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="registry" Jun 08 11:52:09.514750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514729 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" Jun 08 11:52:09.514750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514735 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" Jun 08 11:52:09.514750 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514748 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="feast-apply" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514753 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="feast-apply" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514760 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="feast-init" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514765 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="feast-init" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514775 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514781 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514835 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="ui" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514844 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="online" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514852 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="offline" Jun 08 11:52:09.514925 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.514859 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cccfb3f6-0dac-452f-9537-0b3676b0950e" containerName="registry" Jun 08 11:52:09.519305 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.519286 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.521838 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.521814 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 08 11:52:09.521948 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.521868 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jun 08 11:52:09.521948 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.521915 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 08 11:52:09.521948 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.521907 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 08 11:52:09.522088 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.521943 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 08 11:52:09.522088 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.521952 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-ptkr7\"" Jun 08 11:52:09.522088 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.521868 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 08 11:52:09.522280 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.522262 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 08 11:52:09.530889 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.530871 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct"] Jun 08 11:52:09.561759 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.561731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:52:09.620185 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.620145 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.620185 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.620193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-feast-data\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.620363 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.620217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-registry-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.620363 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.620305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqs85\" (UniqueName: \"kubernetes.io/projected/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-kube-api-access-fqs85\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.620363 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.620358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.620454 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.620408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.721413 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.721384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqs85\" (UniqueName: \"kubernetes.io/projected/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-kube-api-access-fqs85\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.721566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.721446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.721566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.721481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.721566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.721496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.721566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.721516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-feast-data\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.721566 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.721537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-registry-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.721820 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:52:09.721596 2562 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 08 11:52:09.721820 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:52:09.721659 2562 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jun 08 11:52:09.721820 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:52:09.721668 2562 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 08 11:52:09.721820 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:52:09.721724 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls podName:9a5fdf58-8e90-42e0-86f8-e0bb284801c6 nodeName:}" failed. No retries permitted until 2026-06-08 11:52:10.221705997 +0000 UTC m=+789.523482671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls") pod "feast-simple-feast-remote-setup-5674bc7758-k7xct" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 08 11:52:09.721820 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:52:09.721739 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls podName:9a5fdf58-8e90-42e0-86f8-e0bb284801c6 nodeName:}" failed. No retries permitted until 2026-06-08 11:52:10.221733656 +0000 UTC m=+789.523510330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls") pod "feast-simple-feast-remote-setup-5674bc7758-k7xct" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 08 11:52:09.721820 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:52:09.721749 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls podName:9a5fdf58-8e90-42e0-86f8-e0bb284801c6 nodeName:}" failed. No retries permitted until 2026-06-08 11:52:10.221744534 +0000 UTC m=+789.523521208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls") pod "feast-simple-feast-remote-setup-5674bc7758-k7xct" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6") : secret "feast-simple-feast-remote-setup-online-tls" not found Jun 08 11:52:09.722037 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.721932 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-feast-data\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.722183 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.722165 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-registry-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.732529 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.732507 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqs85\" (UniqueName: \"kubernetes.io/projected/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-kube-api-access-fqs85\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:09.846061 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.846019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:52:09.946710 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:09.946683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:52:10.214859 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.214780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:52:10.226947 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.226915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:10.227117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.226957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:10.227117 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.226974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:10.229468 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.229448 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:10.229556 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.229480 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:10.229556 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.229487 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls\") pod \"feast-simple-feast-remote-setup-5674bc7758-k7xct\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:10.428341 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.428301 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:10.563673 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:10.563644 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct"] Jun 08 11:52:10.565190 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:52:10.565158 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a5fdf58_8e90_42e0_86f8_e0bb284801c6.slice/crio-fa3fe48e2f8c10513bc0e4fe8efdc05cd8b9932e2218bc00aef5f833f9ada4a0 WatchSource:0}: Error finding container fa3fe48e2f8c10513bc0e4fe8efdc05cd8b9932e2218bc00aef5f833f9ada4a0: Status 404 returned error can't find the container with id fa3fe48e2f8c10513bc0e4fe8efdc05cd8b9932e2218bc00aef5f833f9ada4a0 Jun 08 11:52:11.226022 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:11.225983 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerStarted","Data":"37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb"} Jun 08 11:52:11.226196 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:11.226040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerStarted","Data":"fa3fe48e2f8c10513bc0e4fe8efdc05cd8b9932e2218bc00aef5f833f9ada4a0"} Jun 08 11:52:15.242927 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:15.242841 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerID="37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb" exitCode=0 Jun 08 11:52:15.242927 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:15.242914 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerDied","Data":"37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb"} Jun 08 11:52:16.248595 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:16.248555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerStarted","Data":"8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2"} Jun 08 11:52:21.269792 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:21.269753 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerID="8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2" exitCode=0 Jun 08 11:52:21.270168 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:21.269827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerDied","Data":"8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2"} Jun 08 11:52:22.279526 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.279486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerStarted","Data":"b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b"} Jun 08 11:52:22.279526 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.279533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerStarted","Data":"782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b"} Jun 08 11:52:22.280085 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.279546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerStarted","Data":"88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49"} Jun 08 11:52:22.302944 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.302887 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podStartSLOduration=13.302865884 podStartE2EDuration="13.302865884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:52:22.301223462 +0000 UTC m=+801.603000159" watchObservedRunningTime="2026-06-08 11:52:22.302865884 +0000 UTC m=+801.604642607" Jun 08 11:52:22.428550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.428459 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:22.428550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.428516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:22.428550 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.428532 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:22.430204 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.430166 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Jun 08 11:52:22.430328 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.430232 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:52:22.430328 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:22.430176 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:52:25.429027 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:25.428974 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:52:25.429410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:25.428981 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Jun 08 11:52:25.429410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:25.428999 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:52:28.429795 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:28.429765 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:28.430193 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:28.429877 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:28.430264 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:28.430235 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:28.430264 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:28.430255 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:28.430633 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:28.430596 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:28.430728 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:28.430714 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:28.433923 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:28.433904 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:29.314054 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:29.314016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:29.316972 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:29.316956 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:52:29.531364 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:52:29.531330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:25.517856 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:25.517824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:30.829189 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.829151 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct"] Jun 08 11:53:30.829786 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.829392 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" containerID="cri-o://88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49" gracePeriod=30 Jun 08 11:53:30.829786 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.829493 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" containerID="cri-o://b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b" gracePeriod=30 Jun 08 11:53:30.829786 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.829583 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="offline" containerID="cri-o://782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b" gracePeriod=30 Jun 08 11:53:30.835085 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.835035 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:30.910556 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.910518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:30.947054 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.947011 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr"] Jun 08 11:53:30.947300 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.947271 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="registry" containerID="cri-o://f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4" gracePeriod=30 Jun 08 11:53:30.947444 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.947404 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="offline" containerID="cri-o://9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8" gracePeriod=30 Jun 08 11:53:30.947549 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.947461 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" containerID="cri-o://6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080" gracePeriod=30 Jun 08 11:53:30.947860 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:30.947691 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" containerID="cri-o://f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92" gracePeriod=30 Jun 08 11:53:31.008986 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:31.008951 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:31.109062 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:31.108935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:31.558427 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:31.558387 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerID="b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b" exitCode=0 Jun 08 11:53:31.558586 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:31.558459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerDied","Data":"b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b"} Jun 08 11:53:32.564975 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:32.564945 2562 generic.go:356] "Generic (PLEG): container finished" podID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerID="f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92" exitCode=0 Jun 08 11:53:32.564975 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:32.564969 2562 generic.go:356] "Generic (PLEG): container finished" podID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerID="6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080" exitCode=0 Jun 08 11:53:32.565467 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:32.565014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerDied","Data":"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92"} Jun 08 11:53:32.565467 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:32.565054 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerDied","Data":"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080"} Jun 08 11:53:32.573080 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:32.573042 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerID="88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49" exitCode=0 Jun 08 11:53:32.573232 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:32.573083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerDied","Data":"88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49"} Jun 08 11:53:33.023508 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:33.023420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:36.110168 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:36.110120 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jun 08 11:53:36.119443 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:36.119416 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:53:36.974825 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:36.974780 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jun 08 11:53:38.430370 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:38.430319 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:53:39.314834 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:39.314789 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:53:46.109718 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:46.109669 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jun 08 11:53:46.974488 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:46.974445 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jun 08 11:53:48.430398 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:48.430354 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:53:49.314492 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:49.314449 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:53:56.110135 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:56.110074 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jun 08 11:53:56.110658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:56.110221 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:53:56.974366 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:56.974316 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jun 08 11:53:56.974547 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:56.974452 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:53:58.431264 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:58.431211 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Jun 08 11:53:58.431763 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:58.431355 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:53:59.314946 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:59.314905 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Jun 08 11:53:59.315118 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:53:59.315024 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:54:01.202528 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.202440 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:54:01.202932 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.202884 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:54:01.483509 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.483479 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:54:01.545053 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545023 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls\") pod \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " Jun 08 11:54:01.545227 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545070 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqs85\" (UniqueName: \"kubernetes.io/projected/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-kube-api-access-fqs85\") pod \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " Jun 08 11:54:01.545227 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545101 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-feast-data\") pod \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " Jun 08 11:54:01.545227 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545143 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-registry-tls\") pod \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " Jun 08 11:54:01.545227 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545207 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls\") pod \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " Jun 08 11:54:01.545425 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545229 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls\") pod \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\" (UID: \"9a5fdf58-8e90-42e0-86f8-e0bb284801c6\") " Jun 08 11:54:01.545580 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545543 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9a5fdf58-8e90-42e0-86f8-e0bb284801c6" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 11:54:01.545749 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.545669 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-feast-data" (OuterVolumeSpecName: "feast-data") pod "9a5fdf58-8e90-42e0-86f8-e0bb284801c6" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:54:01.547603 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.547579 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9a5fdf58-8e90-42e0-86f8-e0bb284801c6" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:54:01.547760 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.547602 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9a5fdf58-8e90-42e0-86f8-e0bb284801c6" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:54:01.547905 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.547882 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls" (OuterVolumeSpecName: "online-tls") pod "9a5fdf58-8e90-42e0-86f8-e0bb284801c6" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:54:01.548006 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.547896 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-kube-api-access-fqs85" (OuterVolumeSpecName: "kube-api-access-fqs85") pod "9a5fdf58-8e90-42e0-86f8-e0bb284801c6" (UID: "9a5fdf58-8e90-42e0-86f8-e0bb284801c6"). InnerVolumeSpecName "kube-api-access-fqs85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:54:01.596059 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.596032 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:54:01.646601 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.646567 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-registry-tls\") pod \"31458c41-68c6-44fc-bafe-863b3d285b9a\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " Jun 08 11:54:01.646601 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.646628 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31458c41-68c6-44fc-bafe-863b3d285b9a-feast-data\") pod \"31458c41-68c6-44fc-bafe-863b3d285b9a\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " Jun 08 11:54:01.646936 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.646658 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-online-tls\") pod \"31458c41-68c6-44fc-bafe-863b3d285b9a\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " Jun 08 11:54:01.646936 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.646773 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-ui-tls\") pod \"31458c41-68c6-44fc-bafe-863b3d285b9a\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " Jun 08 11:54:01.646936 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.646809 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlm92\" (UniqueName: \"kubernetes.io/projected/31458c41-68c6-44fc-bafe-863b3d285b9a-kube-api-access-mlm92\") pod \"31458c41-68c6-44fc-bafe-863b3d285b9a\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " Jun 08 11:54:01.646936 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.646872 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-offline-tls\") pod \"31458c41-68c6-44fc-bafe-863b3d285b9a\" (UID: \"31458c41-68c6-44fc-bafe-863b3d285b9a\") " Jun 08 11:54:01.647174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.647100 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.647174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.647123 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.647174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.647137 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.647174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.647134 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31458c41-68c6-44fc-bafe-863b3d285b9a-feast-data" (OuterVolumeSpecName: "feast-data") pod "31458c41-68c6-44fc-bafe-863b3d285b9a" (UID: "31458c41-68c6-44fc-bafe-863b3d285b9a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:54:01.647174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.647152 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.647174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.647163 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.647174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.647175 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqs85\" (UniqueName: \"kubernetes.io/projected/9a5fdf58-8e90-42e0-86f8-e0bb284801c6-kube-api-access-fqs85\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.648980 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.648947 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-online-tls" (OuterVolumeSpecName: "online-tls") pod "31458c41-68c6-44fc-bafe-863b3d285b9a" (UID: "31458c41-68c6-44fc-bafe-863b3d285b9a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:54:01.648980 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.648965 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "31458c41-68c6-44fc-bafe-863b3d285b9a" (UID: "31458c41-68c6-44fc-bafe-863b3d285b9a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:54:01.649114 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.649043 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "31458c41-68c6-44fc-bafe-863b3d285b9a" (UID: "31458c41-68c6-44fc-bafe-863b3d285b9a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:54:01.649489 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.649467 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "31458c41-68c6-44fc-bafe-863b3d285b9a" (UID: "31458c41-68c6-44fc-bafe-863b3d285b9a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:54:01.649554 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.649469 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31458c41-68c6-44fc-bafe-863b3d285b9a-kube-api-access-mlm92" (OuterVolumeSpecName: "kube-api-access-mlm92") pod "31458c41-68c6-44fc-bafe-863b3d285b9a" (UID: "31458c41-68c6-44fc-bafe-863b3d285b9a"). InnerVolumeSpecName "kube-api-access-mlm92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:54:01.688761 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.688727 2562 generic.go:356] "Generic (PLEG): container finished" podID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerID="9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8" exitCode=137 Jun 08 11:54:01.688761 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.688753 2562 generic.go:356] "Generic (PLEG): container finished" podID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerID="f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4" exitCode=137 Jun 08 11:54:01.688984 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.688791 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" Jun 08 11:54:01.688984 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.688806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerDied","Data":"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8"} Jun 08 11:54:01.688984 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.688856 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerDied","Data":"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4"} Jun 08 11:54:01.688984 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.688877 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr" event={"ID":"31458c41-68c6-44fc-bafe-863b3d285b9a","Type":"ContainerDied","Data":"a1eb59d629a9888cb224f70e102a28a50d41a7a101f3d77a673a0f7fd42967b3"} Jun 08 11:54:01.688984 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.688901 2562 scope.go:117] "RemoveContainer" containerID="f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92" Jun 08 11:54:01.691591 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.691564 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerID="782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b" exitCode=137 Jun 08 11:54:01.691726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.691647 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" Jun 08 11:54:01.691726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.691652 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerDied","Data":"782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b"} Jun 08 11:54:01.691726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.691698 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct" event={"ID":"9a5fdf58-8e90-42e0-86f8-e0bb284801c6","Type":"ContainerDied","Data":"fa3fe48e2f8c10513bc0e4fe8efdc05cd8b9932e2218bc00aef5f833f9ada4a0"} Jun 08 11:54:01.699123 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.699100 2562 scope.go:117] "RemoveContainer" containerID="9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8" Jun 08 11:54:01.706769 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.706746 2562 scope.go:117] "RemoveContainer" containerID="6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080" Jun 08 11:54:01.714748 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.714593 2562 scope.go:117] "RemoveContainer" containerID="f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4" Jun 08 11:54:01.716119 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.716099 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr"] Jun 08 11:54:01.718986 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.718966 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7fb8f4b78b-l5zhr"] Jun 08 11:54:01.723065 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.723043 2562 scope.go:117] "RemoveContainer" containerID="c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9" Jun 08 11:54:01.730469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.730404 2562 scope.go:117] "RemoveContainer" containerID="f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3" Jun 08 11:54:01.732523 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.732503 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct"] Jun 08 11:54:01.735116 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.735092 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5674bc7758-k7xct"] Jun 08 11:54:01.738790 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.738776 2562 scope.go:117] "RemoveContainer" containerID="f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92" Jun 08 11:54:01.739125 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.739102 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92\": container with ID starting with f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92 not found: ID does not exist" containerID="f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92" Jun 08 11:54:01.739191 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.739138 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92"} err="failed to get container status \"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92\": rpc error: code = NotFound desc = could not find container \"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92\": container with ID starting with f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92 not found: ID does not exist" Jun 08 11:54:01.739191 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.739166 2562 scope.go:117] "RemoveContainer" containerID="9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8" Jun 08 11:54:01.739464 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.739445 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8\": container with ID starting with 9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8 not found: ID does not exist" containerID="9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8" Jun 08 11:54:01.739504 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.739471 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8"} err="failed to get container status \"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8\": rpc error: code = NotFound desc = could not find container \"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8\": container with ID starting with 9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8 not found: ID does not exist" Jun 08 11:54:01.739504 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.739486 2562 scope.go:117] "RemoveContainer" containerID="6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080" Jun 08 11:54:01.739733 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.739718 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080\": container with ID starting with 6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080 not found: ID does not exist" containerID="6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080" Jun 08 11:54:01.739789 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.739736 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080"} err="failed to get container status \"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080\": rpc error: code = NotFound desc = could not find container \"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080\": container with ID starting with 6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080 not found: ID does not exist" Jun 08 11:54:01.739789 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.739750 2562 scope.go:117] "RemoveContainer" containerID="f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4" Jun 08 11:54:01.739985 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.739958 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4\": container with ID starting with f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4 not found: ID does not exist" containerID="f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4" Jun 08 11:54:01.740025 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.739989 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4"} err="failed to get container status \"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4\": rpc error: code = NotFound desc = could not find container \"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4\": container with ID starting with f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4 not found: ID does not exist" Jun 08 11:54:01.740025 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740004 2562 scope.go:117] "RemoveContainer" containerID="c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9" Jun 08 11:54:01.740243 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.740226 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9\": container with ID starting with c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9 not found: ID does not exist" containerID="c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9" Jun 08 11:54:01.740284 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740247 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9"} err="failed to get container status \"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9\": rpc error: code = NotFound desc = could not find container \"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9\": container with ID starting with c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9 not found: ID does not exist" Jun 08 11:54:01.740284 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740262 2562 scope.go:117] "RemoveContainer" containerID="f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3" Jun 08 11:54:01.740448 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.740434 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3\": container with ID starting with f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3 not found: ID does not exist" containerID="f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3" Jun 08 11:54:01.740487 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740454 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3"} err="failed to get container status \"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3\": rpc error: code = NotFound desc = could not find container \"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3\": container with ID starting with f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3 not found: ID does not exist" Jun 08 11:54:01.740487 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740465 2562 scope.go:117] "RemoveContainer" containerID="f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92" Jun 08 11:54:01.740661 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740644 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92"} err="failed to get container status \"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92\": rpc error: code = NotFound desc = could not find container \"f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92\": container with ID starting with f8a4f839987fa3ef63e11fe026690f44c2ba58a2d61f42d3a41911d78d083c92 not found: ID does not exist" Jun 08 11:54:01.740708 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740662 2562 scope.go:117] "RemoveContainer" containerID="9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8" Jun 08 11:54:01.740847 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740832 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8"} err="failed to get container status \"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8\": rpc error: code = NotFound desc = could not find container \"9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8\": container with ID starting with 9339f5964fd1f02e0dfc8c31acb76b7bd6b63de8fd6a3c65fb05f82663f0f0f8 not found: ID does not exist" Jun 08 11:54:01.740894 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.740848 2562 scope.go:117] "RemoveContainer" containerID="6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080" Jun 08 11:54:01.741069 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741052 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080"} err="failed to get container status \"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080\": rpc error: code = NotFound desc = could not find container \"6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080\": container with ID starting with 6c453779d301da51469a75873bc5d73bbc4fc9de28a7d0c7667bfdfa57d46080 not found: ID does not exist" Jun 08 11:54:01.741069 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741067 2562 scope.go:117] "RemoveContainer" containerID="f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4" Jun 08 11:54:01.741255 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741234 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4"} err="failed to get container status \"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4\": rpc error: code = NotFound desc = could not find container \"f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4\": container with ID starting with f15d667b5ed6577bed8d7eeb2612ef860b02784eb593000dddc8aea3a66741c4 not found: ID does not exist" Jun 08 11:54:01.741294 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741257 2562 scope.go:117] "RemoveContainer" containerID="c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9" Jun 08 11:54:01.741482 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741465 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9"} err="failed to get container status \"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9\": rpc error: code = NotFound desc = could not find container \"c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9\": container with ID starting with c7ba0133a7f888dc2b9ae16b2d57fc30d8520f7ac245c3fd1bd008eba2db8ff9 not found: ID does not exist" Jun 08 11:54:01.741521 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741483 2562 scope.go:117] "RemoveContainer" containerID="f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3" Jun 08 11:54:01.741701 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741683 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3"} err="failed to get container status \"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3\": rpc error: code = NotFound desc = could not find container \"f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3\": container with ID starting with f1a69abdbf94ff0f3f18bed52f0fbbeab67b2553738cbff933a2a51b37a758d3 not found: ID does not exist" Jun 08 11:54:01.741749 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.741700 2562 scope.go:117] "RemoveContainer" containerID="b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b" Jun 08 11:54:01.747922 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.747906 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.748007 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.747927 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlm92\" (UniqueName: \"kubernetes.io/projected/31458c41-68c6-44fc-bafe-863b3d285b9a-kube-api-access-mlm92\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.748007 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.747940 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.748007 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.747949 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.748007 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.747975 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/31458c41-68c6-44fc-bafe-863b3d285b9a-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.748007 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.747983 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/31458c41-68c6-44fc-bafe-863b3d285b9a-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:54:01.749514 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.749498 2562 scope.go:117] "RemoveContainer" containerID="782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b" Jun 08 11:54:01.756916 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.756897 2562 scope.go:117] "RemoveContainer" containerID="88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49" Jun 08 11:54:01.764032 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.764008 2562 scope.go:117] "RemoveContainer" containerID="8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2" Jun 08 11:54:01.772891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.772865 2562 scope.go:117] "RemoveContainer" containerID="37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb" Jun 08 11:54:01.780517 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.780499 2562 scope.go:117] "RemoveContainer" containerID="b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b" Jun 08 11:54:01.780815 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.780788 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b\": container with ID starting with b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b not found: ID does not exist" containerID="b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b" Jun 08 11:54:01.780864 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.780820 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b"} err="failed to get container status \"b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b\": rpc error: code = NotFound desc = could not find container \"b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b\": container with ID starting with b065e16debc3e4226adb12cc16adcc033eb328b951effc7116fc754d25a8337b not found: ID does not exist" Jun 08 11:54:01.780864 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.780838 2562 scope.go:117] "RemoveContainer" containerID="782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b" Jun 08 11:54:01.781078 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.781064 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b\": container with ID starting with 782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b not found: ID does not exist" containerID="782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b" Jun 08 11:54:01.781118 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.781080 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b"} err="failed to get container status \"782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b\": rpc error: code = NotFound desc = could not find container \"782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b\": container with ID starting with 782ba625412af4241e7383c9c723ec939d08aeafff38ba5dc60d530094151f0b not found: ID does not exist" Jun 08 11:54:01.781118 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.781096 2562 scope.go:117] "RemoveContainer" containerID="88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49" Jun 08 11:54:01.781308 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.781289 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49\": container with ID starting with 88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49 not found: ID does not exist" containerID="88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49" Jun 08 11:54:01.781376 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.781318 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49"} err="failed to get container status \"88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49\": rpc error: code = NotFound desc = could not find container \"88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49\": container with ID starting with 88522676d1224209a83d1429b9ee026a89d45b8b445b1baa1c6add2eaf831c49 not found: ID does not exist" Jun 08 11:54:01.781376 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.781339 2562 scope.go:117] "RemoveContainer" containerID="8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2" Jun 08 11:54:01.781565 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.781549 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2\": container with ID starting with 8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2 not found: ID does not exist" containerID="8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2" Jun 08 11:54:01.781629 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.781570 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2"} err="failed to get container status \"8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2\": rpc error: code = NotFound desc = could not find container \"8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2\": container with ID starting with 8f4dee793955593a9c1b21c150c08dd2332257f60cd59ad6514be14f4676aeb2 not found: ID does not exist" Jun 08 11:54:01.781629 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.781583 2562 scope.go:117] "RemoveContainer" containerID="37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb" Jun 08 11:54:01.781827 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:01.781812 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb\": container with ID starting with 37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb not found: ID does not exist" containerID="37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb" Jun 08 11:54:01.781870 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:01.781831 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb"} err="failed to get container status \"37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb\": rpc error: code = NotFound desc = could not find container \"37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb\": container with ID starting with 37f888d8a9b61b68993145dc07cc2cebb51af9aba9297963c6b95ddac5568dcb not found: ID does not exist" Jun 08 11:54:03.203223 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:03.203191 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" path="/var/lib/kubelet/pods/31458c41-68c6-44fc-bafe-863b3d285b9a/volumes" Jun 08 11:54:03.203813 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:03.203799 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" path="/var/lib/kubelet/pods/9a5fdf58-8e90-42e0-86f8-e0bb284801c6/volumes" Jun 08 11:54:13.595541 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.595505 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:54:13.831013 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.830979 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-mvzfq"] Jun 08 11:54:13.831270 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831258 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831271 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831286 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="feast-init" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831292 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="feast-init" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831301 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831307 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831315 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="feast-apply" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831320 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="feast-apply" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831328 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="registry" Jun 08 11:54:13.831332 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831334 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="registry" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831341 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="offline" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831346 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="offline" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831353 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="feast-apply" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831358 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="feast-apply" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831363 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831368 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831375 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831379 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831387 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="feast-init" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831392 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="feast-init" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831396 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="offline" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831401 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="offline" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831460 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="online" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831466 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="registry" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831472 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="ui" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831480 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="31458c41-68c6-44fc-bafe-863b3d285b9a" containerName="offline" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831485 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="online" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831490 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="ui" Jun 08 11:54:13.831654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.831494 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5fdf58-8e90-42e0-86f8-e0bb284801c6" containerName="offline" Jun 08 11:54:13.834284 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.834267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:13.836444 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.836415 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-597hb\"" Jun 08 11:54:13.836654 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.836633 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jun 08 11:54:13.837096 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.837077 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 11:54:13.837178 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.837082 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 11:54:13.843273 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.843250 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-mvzfq"] Jun 08 11:54:13.866479 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.866389 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-zlkvt"] Jun 08 11:54:13.870069 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.870045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-zlkvt" Jun 08 11:54:13.879016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.878984 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-zlkvt"] Jun 08 11:54:13.945838 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.945796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttpc\" (UniqueName: \"kubernetes.io/projected/78a2b4bb-bf3d-40e9-982a-e0f090a46cd5-kube-api-access-bttpc\") pod \"redis-69bbd798b6-zlkvt\" (UID: \"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5\") " pod="test-ns-feast/redis-69bbd798b6-zlkvt" Jun 08 11:54:13.946053 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.945848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-postgresdata\") pod \"postgres-864846d4f5-mvzfq\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:13.946053 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:13.945911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvh2f\" (UniqueName: \"kubernetes.io/projected/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-kube-api-access-gvh2f\") pod \"postgres-864846d4f5-mvzfq\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:14.046679 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.046641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttpc\" (UniqueName: \"kubernetes.io/projected/78a2b4bb-bf3d-40e9-982a-e0f090a46cd5-kube-api-access-bttpc\") pod \"redis-69bbd798b6-zlkvt\" (UID: \"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5\") " pod="test-ns-feast/redis-69bbd798b6-zlkvt" Jun 08 11:54:14.046847 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.046687 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-postgresdata\") pod \"postgres-864846d4f5-mvzfq\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:14.046929 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.046890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvh2f\" (UniqueName: \"kubernetes.io/projected/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-kube-api-access-gvh2f\") pod \"postgres-864846d4f5-mvzfq\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:14.047083 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.047067 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-postgresdata\") pod \"postgres-864846d4f5-mvzfq\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:14.055421 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.055391 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvh2f\" (UniqueName: \"kubernetes.io/projected/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-kube-api-access-gvh2f\") pod \"postgres-864846d4f5-mvzfq\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:14.055558 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.055541 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttpc\" (UniqueName: \"kubernetes.io/projected/78a2b4bb-bf3d-40e9-982a-e0f090a46cd5-kube-api-access-bttpc\") pod \"redis-69bbd798b6-zlkvt\" (UID: \"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5\") " pod="test-ns-feast/redis-69bbd798b6-zlkvt" Jun 08 11:54:14.162300 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.162204 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:54:14.179409 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.179380 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-zlkvt" Jun 08 11:54:14.304148 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.304005 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-mvzfq"] Jun 08 11:54:14.307201 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:54:14.307170 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8774cb9_f1d9_48a9_92db_12bfa7a4c2fe.slice/crio-a9122c0a187feff89e0bf6017a2691dcb3f7356fb9949f82712c28566243dde0 WatchSource:0}: Error finding container a9122c0a187feff89e0bf6017a2691dcb3f7356fb9949f82712c28566243dde0: Status 404 returned error can't find the container with id a9122c0a187feff89e0bf6017a2691dcb3f7356fb9949f82712c28566243dde0 Jun 08 11:54:14.309224 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.309204 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 11:54:14.326887 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.326858 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-zlkvt"] Jun 08 11:54:14.330417 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:54:14.330391 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78a2b4bb_bf3d_40e9_982a_e0f090a46cd5.slice/crio-ed4a4eac36be82005f130743eeea16d913723ca5e59bfd3844483e669c68a4cc WatchSource:0}: Error finding container ed4a4eac36be82005f130743eeea16d913723ca5e59bfd3844483e669c68a4cc: Status 404 returned error can't find the container with id ed4a4eac36be82005f130743eeea16d913723ca5e59bfd3844483e669c68a4cc Jun 08 11:54:14.741562 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.741522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-mvzfq" event={"ID":"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe","Type":"ContainerStarted","Data":"a9122c0a187feff89e0bf6017a2691dcb3f7356fb9949f82712c28566243dde0"} Jun 08 11:54:14.742912 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:14.742878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-zlkvt" event={"ID":"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5","Type":"ContainerStarted","Data":"ed4a4eac36be82005f130743eeea16d913723ca5e59bfd3844483e669c68a4cc"} Jun 08 11:54:16.757338 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:16.757297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-zlkvt" event={"ID":"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5","Type":"ContainerStarted","Data":"50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315"} Jun 08 11:54:16.772026 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:16.771967 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-zlkvt" podStartSLOduration=2.094230203 podStartE2EDuration="3.771949315s" totalImagesPullingTime="1.677719112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:54:16.770911431 +0000 UTC m=+916.072688142" watchObservedRunningTime="2026-06-08 11:54:16.771949315 +0000 UTC m=+916.073726042" Jun 08 11:54:19.620396 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:19.620358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:54:19.776674 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:19.776575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-mvzfq" event={"ID":"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe","Type":"ContainerStarted","Data":"c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af"} Jun 08 11:54:19.792105 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:19.792054 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-mvzfq" podStartSLOduration=1.612372452 podStartE2EDuration="6.792038407s" totalImagesPullingTime="5.179665955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:54:19.791082034 +0000 UTC m=+919.092858762" watchObservedRunningTime="2026-06-08 11:54:19.792038407 +0000 UTC m=+919.093815104" Jun 08 11:54:22.398135 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.398074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:54:22.498172 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.498138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:54:22.576205 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.576172 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw"] Jun 08 11:54:22.580336 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.580319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.582653 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.582627 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jun 08 11:54:22.582777 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.582662 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-sjxjf\"" Jun 08 11:54:22.582777 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.582752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jun 08 11:54:22.582891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.582774 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jun 08 11:54:22.582980 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.582967 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jun 08 11:54:22.591505 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.591484 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw"] Jun 08 11:54:22.714489 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.714414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:54:22.726066 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.726041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg5jm\" (UniqueName: \"kubernetes.io/projected/b8c74841-031f-4068-b784-227ac68d1c37-kube-api-access-hg5jm\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.726125 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.726075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-registry-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.726125 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.726106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c74841-031f-4068-b784-227ac68d1c37-feast-data\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.726199 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.726128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.726235 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.726203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-online-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.815530 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.815500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:54:22.827019 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.826994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c74841-031f-4068-b784-227ac68d1c37-feast-data\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.827154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.827025 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.827154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.827056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-online-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.827154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.827088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg5jm\" (UniqueName: \"kubernetes.io/projected/b8c74841-031f-4068-b784-227ac68d1c37-kube-api-access-hg5jm\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.827154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.827108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-registry-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.827154 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:22.827133 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Jun 08 11:54:22.827406 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:54:22.827184 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls podName:b8c74841-031f-4068-b784-227ac68d1c37 nodeName:}" failed. No retries permitted until 2026-06-08 11:54:23.327169498 +0000 UTC m=+922.628946173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls") pod "feast-credit-scoring-5f8b698bb7-d5wgw" (UID: "b8c74841-031f-4068-b784-227ac68d1c37") : secret "feast-credit-scoring-offline-tls" not found Jun 08 11:54:22.827406 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.827370 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c74841-031f-4068-b784-227ac68d1c37-feast-data\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.829689 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.829667 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-online-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.829832 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.829807 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-registry-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.837769 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.837749 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg5jm\" (UniqueName: \"kubernetes.io/projected/b8c74841-031f-4068-b784-227ac68d1c37-kube-api-access-hg5jm\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:22.987149 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:22.987065 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:54:23.332113 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:23.332077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:23.334875 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:23.334843 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls\") pod \"feast-credit-scoring-5f8b698bb7-d5wgw\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:23.490261 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:23.490222 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:23.627745 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:23.627718 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw"] Jun 08 11:54:23.629283 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:54:23.629256 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c74841_031f_4068_b784_227ac68d1c37.slice/crio-e0bb6a3d2a3784fe1e4cf71353895d82a9a94c948a9d09032a3062fe43b8d65c WatchSource:0}: Error finding container e0bb6a3d2a3784fe1e4cf71353895d82a9a94c948a9d09032a3062fe43b8d65c: Status 404 returned error can't find the container with id e0bb6a3d2a3784fe1e4cf71353895d82a9a94c948a9d09032a3062fe43b8d65c Jun 08 11:54:23.793890 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:23.793853 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerStarted","Data":"1a74470c0c835ea767b093a21f146ae5dfeb80eb13ba64f789dda3bd4251a732"} Jun 08 11:54:23.794054 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:23.793916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerStarted","Data":"e0bb6a3d2a3784fe1e4cf71353895d82a9a94c948a9d09032a3062fe43b8d65c"} Jun 08 11:54:25.803978 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:25.803944 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c74841-031f-4068-b784-227ac68d1c37" containerID="1a74470c0c835ea767b093a21f146ae5dfeb80eb13ba64f789dda3bd4251a732" exitCode=0 Jun 08 11:54:25.804380 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:25.804015 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerDied","Data":"1a74470c0c835ea767b093a21f146ae5dfeb80eb13ba64f789dda3bd4251a732"} Jun 08 11:54:26.809538 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:26.809503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerStarted","Data":"3526a35cc462ad654c742250983d068c87198fcb02ca1a7c2691a020ab1295cb"} Jun 08 11:54:32.834773 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:32.834735 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c74841-031f-4068-b784-227ac68d1c37" containerID="3526a35cc462ad654c742250983d068c87198fcb02ca1a7c2691a020ab1295cb" exitCode=0 Jun 08 11:54:32.835234 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:32.834807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerDied","Data":"3526a35cc462ad654c742250983d068c87198fcb02ca1a7c2691a020ab1295cb"} Jun 08 11:54:33.851180 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:33.851133 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerStarted","Data":"1100e48ca2dd31d955f35a49636972fbccca95d7ab6baeee099f12cc3909f40f"} Jun 08 11:54:33.851583 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:33.851249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerStarted","Data":"096cf52b188237fc5aba504fcca0cfcd6dfaf922faf9875af4e61c45ee446bef"} Jun 08 11:54:33.851583 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:33.851278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerStarted","Data":"127c3a9b40eac248d4da5d416c3be976ec161667c85bb5746b846ecb454beb3f"} Jun 08 11:54:33.880387 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:33.880323 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podStartSLOduration=11.880300885 podStartE2EDuration="11.880300885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:54:33.878263175 +0000 UTC m=+933.180039872" watchObservedRunningTime="2026-06-08 11:54:33.880300885 +0000 UTC m=+933.182077582" Jun 08 11:54:35.491124 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:35.491078 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:35.491676 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:35.491173 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:35.491939 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:35.491905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:35.492824 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:35.492793 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" Jun 08 11:54:35.492945 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:35.492815 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.28:8816: connect: connection refused" Jun 08 11:54:35.492945 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:35.492844 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jun 08 11:54:38.491318 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:38.491264 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" Jun 08 11:54:38.492513 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:38.492483 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:38.493104 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:38.493075 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:38.493775 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:38.493754 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:38.813185 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:38.813152 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:38.875981 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:38.875944 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:38.880302 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:38.880281 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:41.490901 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:41.490857 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" Jun 08 11:54:43.490729 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:43.490688 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:44.491188 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:44.491161 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:44.491880 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:44.491861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:54:44.768405 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:54:44.768325 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:55:39.050110 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:39.050078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:55:44.027203 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.027167 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw"] Jun 08 11:55:44.027714 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.027429 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" containerID="cri-o://127c3a9b40eac248d4da5d416c3be976ec161667c85bb5746b846ecb454beb3f" gracePeriod=30 Jun 08 11:55:44.027714 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.027483 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" containerID="cri-o://096cf52b188237fc5aba504fcca0cfcd6dfaf922faf9875af4e61c45ee446bef" gracePeriod=30 Jun 08 11:55:44.027714 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.027492 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="offline" containerID="cri-o://1100e48ca2dd31d955f35a49636972fbccca95d7ab6baeee099f12cc3909f40f" gracePeriod=30 Jun 08 11:55:44.047667 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.047642 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-mvzfq"] Jun 08 11:55:44.047808 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.047791 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-mvzfq" podUID="c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" containerName="postgres" containerID="cri-o://c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af" gracePeriod=30 Jun 08 11:55:44.056219 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.056190 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-zlkvt"] Jun 08 11:55:44.056659 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.056391 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-zlkvt" podUID="78a2b4bb-bf3d-40e9-982a-e0f090a46cd5" containerName="redis" containerID="cri-o://50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315" gracePeriod=30 Jun 08 11:55:44.111352 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.111331 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:55:44.288738 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.288720 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:55:44.322879 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.322859 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-zlkvt" Jun 08 11:55:44.384505 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.384476 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-postgresdata\") pod \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " Jun 08 11:55:44.384672 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.384511 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvh2f\" (UniqueName: \"kubernetes.io/projected/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-kube-api-access-gvh2f\") pod \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\" (UID: \"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe\") " Jun 08 11:55:44.386548 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.386522 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-kube-api-access-gvh2f" (OuterVolumeSpecName: "kube-api-access-gvh2f") pod "c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" (UID: "c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe"). InnerVolumeSpecName "kube-api-access-gvh2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:55:44.412072 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.412043 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" (UID: "c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:55:44.485041 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.485009 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bttpc\" (UniqueName: \"kubernetes.io/projected/78a2b4bb-bf3d-40e9-982a-e0f090a46cd5-kube-api-access-bttpc\") pod \"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5\" (UID: \"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5\") " Jun 08 11:55:44.485264 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.485247 2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-postgresdata\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:55:44.485320 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.485270 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvh2f\" (UniqueName: \"kubernetes.io/projected/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe-kube-api-access-gvh2f\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:55:44.487107 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.487085 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a2b4bb-bf3d-40e9-982a-e0f090a46cd5-kube-api-access-bttpc" (OuterVolumeSpecName: "kube-api-access-bttpc") pod "78a2b4bb-bf3d-40e9-982a-e0f090a46cd5" (UID: "78a2b4bb-bf3d-40e9-982a-e0f090a46cd5"). InnerVolumeSpecName "kube-api-access-bttpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:55:44.586446 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:44.586356 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttpc\" (UniqueName: \"kubernetes.io/projected/78a2b4bb-bf3d-40e9-982a-e0f090a46cd5-kube-api-access-bttpc\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:55:45.145802 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.145728 2562 generic.go:356] "Generic (PLEG): container finished" podID="78a2b4bb-bf3d-40e9-982a-e0f090a46cd5" containerID="50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315" exitCode=0 Jun 08 11:55:45.145802 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.145768 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-zlkvt" Jun 08 11:55:45.146213 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.145806 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-zlkvt" event={"ID":"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5","Type":"ContainerDied","Data":"50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315"} Jun 08 11:55:45.146213 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.145851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-zlkvt" event={"ID":"78a2b4bb-bf3d-40e9-982a-e0f090a46cd5","Type":"ContainerDied","Data":"ed4a4eac36be82005f130743eeea16d913723ca5e59bfd3844483e669c68a4cc"} Jun 08 11:55:45.146213 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.145875 2562 scope.go:117] "RemoveContainer" containerID="50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315" Jun 08 11:55:45.148770 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.148746 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c74841-031f-4068-b784-227ac68d1c37" containerID="096cf52b188237fc5aba504fcca0cfcd6dfaf922faf9875af4e61c45ee446bef" exitCode=0 Jun 08 11:55:45.148881 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.148792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerDied","Data":"096cf52b188237fc5aba504fcca0cfcd6dfaf922faf9875af4e61c45ee446bef"} Jun 08 11:55:45.150165 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.150144 2562 generic.go:356] "Generic (PLEG): container finished" podID="c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" containerID="c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af" exitCode=0 Jun 08 11:55:45.150268 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.150182 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-mvzfq" event={"ID":"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe","Type":"ContainerDied","Data":"c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af"} Jun 08 11:55:45.150268 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.150188 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-mvzfq" Jun 08 11:55:45.150268 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.150205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-mvzfq" event={"ID":"c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe","Type":"ContainerDied","Data":"a9122c0a187feff89e0bf6017a2691dcb3f7356fb9949f82712c28566243dde0"} Jun 08 11:55:45.154593 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.154574 2562 scope.go:117] "RemoveContainer" containerID="50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315" Jun 08 11:55:45.154887 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:55:45.154866 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315\": container with ID starting with 50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315 not found: ID does not exist" containerID="50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315" Jun 08 11:55:45.154947 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.154894 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315"} err="failed to get container status \"50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315\": rpc error: code = NotFound desc = could not find container \"50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315\": container with ID starting with 50d229844a70f5926de4a207cc93435a889d8e384acd441d10af8bf95d3dd315 not found: ID does not exist" Jun 08 11:55:45.154947 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.154909 2562 scope.go:117] "RemoveContainer" containerID="c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af" Jun 08 11:55:45.162133 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.162115 2562 scope.go:117] "RemoveContainer" containerID="c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af" Jun 08 11:55:45.162386 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:55:45.162368 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af\": container with ID starting with c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af not found: ID does not exist" containerID="c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af" Jun 08 11:55:45.162452 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.162391 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af"} err="failed to get container status \"c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af\": rpc error: code = NotFound desc = could not find container \"c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af\": container with ID starting with c2cb1f8fd7e2ea5469977f3c450e1cd18a5b70ed85cd5ad1b40f4b59794ce6af not found: ID does not exist" Jun 08 11:55:45.165832 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.165813 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-zlkvt"] Jun 08 11:55:45.169067 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.169047 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-zlkvt"] Jun 08 11:55:45.178015 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.177996 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-mvzfq"] Jun 08 11:55:45.181285 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.181264 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-mvzfq"] Jun 08 11:55:45.202370 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.202351 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a2b4bb-bf3d-40e9-982a-e0f090a46cd5" path="/var/lib/kubelet/pods/78a2b4bb-bf3d-40e9-982a-e0f090a46cd5/volumes" Jun 08 11:55:45.202719 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:45.202706 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" path="/var/lib/kubelet/pods/c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe/volumes" Jun 08 11:55:48.876790 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:48.876751 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jun 08 11:55:58.876291 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:55:58.876248 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jun 08 11:56:08.876422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:08.876375 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Jun 08 11:56:08.876911 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:08.876520 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:56:14.262147 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.262110 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c74841-031f-4068-b784-227ac68d1c37" containerID="1100e48ca2dd31d955f35a49636972fbccca95d7ab6baeee099f12cc3909f40f" exitCode=137 Jun 08 11:56:14.262147 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.262142 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8c74841-031f-4068-b784-227ac68d1c37" containerID="127c3a9b40eac248d4da5d416c3be976ec161667c85bb5746b846ecb454beb3f" exitCode=137 Jun 08 11:56:14.262638 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.262197 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerDied","Data":"1100e48ca2dd31d955f35a49636972fbccca95d7ab6baeee099f12cc3909f40f"} Jun 08 11:56:14.262638 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.262255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerDied","Data":"127c3a9b40eac248d4da5d416c3be976ec161667c85bb5746b846ecb454beb3f"} Jun 08 11:56:14.492573 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.492520 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" Jun 08 11:56:14.674623 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.674591 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:56:14.714458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.714426 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-registry-tls\") pod \"b8c74841-031f-4068-b784-227ac68d1c37\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " Jun 08 11:56:14.714458 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.714459 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls\") pod \"b8c74841-031f-4068-b784-227ac68d1c37\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " Jun 08 11:56:14.714815 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.714489 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-online-tls\") pod \"b8c74841-031f-4068-b784-227ac68d1c37\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " Jun 08 11:56:14.714815 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.714516 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg5jm\" (UniqueName: \"kubernetes.io/projected/b8c74841-031f-4068-b784-227ac68d1c37-kube-api-access-hg5jm\") pod \"b8c74841-031f-4068-b784-227ac68d1c37\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " Jun 08 11:56:14.714815 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.714537 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c74841-031f-4068-b784-227ac68d1c37-feast-data\") pod \"b8c74841-031f-4068-b784-227ac68d1c37\" (UID: \"b8c74841-031f-4068-b784-227ac68d1c37\") " Jun 08 11:56:14.716825 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.716781 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c74841-031f-4068-b784-227ac68d1c37-kube-api-access-hg5jm" (OuterVolumeSpecName: "kube-api-access-hg5jm") pod "b8c74841-031f-4068-b784-227ac68d1c37" (UID: "b8c74841-031f-4068-b784-227ac68d1c37"). InnerVolumeSpecName "kube-api-access-hg5jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 11:56:14.717045 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.717026 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b8c74841-031f-4068-b784-227ac68d1c37" (UID: "b8c74841-031f-4068-b784-227ac68d1c37"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:56:14.717158 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.717141 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b8c74841-031f-4068-b784-227ac68d1c37" (UID: "b8c74841-031f-4068-b784-227ac68d1c37"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:56:14.717292 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.717266 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-online-tls" (OuterVolumeSpecName: "online-tls") pod "b8c74841-031f-4068-b784-227ac68d1c37" (UID: "b8c74841-031f-4068-b784-227ac68d1c37"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 11:56:14.722768 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.722736 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c74841-031f-4068-b784-227ac68d1c37-feast-data" (OuterVolumeSpecName: "feast-data") pod "b8c74841-031f-4068-b784-227ac68d1c37" (UID: "b8c74841-031f-4068-b784-227ac68d1c37"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 11:56:14.815476 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.815441 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:56:14.815476 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.815473 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg5jm\" (UniqueName: \"kubernetes.io/projected/b8c74841-031f-4068-b784-227ac68d1c37-kube-api-access-hg5jm\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:56:14.815476 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.815482 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c74841-031f-4068-b784-227ac68d1c37-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:56:14.815846 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.815491 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:56:14.815846 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:14.815498 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b8c74841-031f-4068-b784-227ac68d1c37-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 11:56:15.268474 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.268438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" event={"ID":"b8c74841-031f-4068-b784-227ac68d1c37","Type":"ContainerDied","Data":"e0bb6a3d2a3784fe1e4cf71353895d82a9a94c948a9d09032a3062fe43b8d65c"} Jun 08 11:56:15.268936 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.268495 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw" Jun 08 11:56:15.268936 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.268512 2562 scope.go:117] "RemoveContainer" containerID="1100e48ca2dd31d955f35a49636972fbccca95d7ab6baeee099f12cc3909f40f" Jun 08 11:56:15.277394 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.277376 2562 scope.go:117] "RemoveContainer" containerID="096cf52b188237fc5aba504fcca0cfcd6dfaf922faf9875af4e61c45ee446bef" Jun 08 11:56:15.285896 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.285845 2562 scope.go:117] "RemoveContainer" containerID="127c3a9b40eac248d4da5d416c3be976ec161667c85bb5746b846ecb454beb3f" Jun 08 11:56:15.288391 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.287645 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw"] Jun 08 11:56:15.294385 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.294353 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-5f8b698bb7-d5wgw"] Jun 08 11:56:15.300469 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.300448 2562 scope.go:117] "RemoveContainer" containerID="3526a35cc462ad654c742250983d068c87198fcb02ca1a7c2691a020ab1295cb" Jun 08 11:56:15.307991 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:15.307971 2562 scope.go:117] "RemoveContainer" containerID="1a74470c0c835ea767b093a21f146ae5dfeb80eb13ba64f789dda3bd4251a732" Jun 08 11:56:17.202931 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:17.202897 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c74841-031f-4068-b784-227ac68d1c37" path="/var/lib/kubelet/pods/b8c74841-031f-4068-b784-227ac68d1c37/volumes" Jun 08 11:56:21.334900 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:21.334870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:56:21.434355 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:21.434324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:56:21.960532 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:21.960500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:56:22.023525 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:22.023489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:56:22.105549 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:56:22.105515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:57:20.298704 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:57:20.298647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:00.001866 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.001827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:00.102625 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.102581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:00.238178 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238146 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-t7d6j"] Jun 08 11:58:00.238444 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238432 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" containerName="postgres" Jun 08 11:58:00.238491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238446 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" containerName="postgres" Jun 08 11:58:00.238491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238454 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="feast-apply" Jun 08 11:58:00.238491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238460 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="feast-apply" Jun 08 11:58:00.238491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238467 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" Jun 08 11:58:00.238491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238473 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" Jun 08 11:58:00.238491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238481 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a2b4bb-bf3d-40e9-982a-e0f090a46cd5" containerName="redis" Jun 08 11:58:00.238491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238486 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a2b4bb-bf3d-40e9-982a-e0f090a46cd5" containerName="redis" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238499 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="offline" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238504 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="offline" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238516 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238521 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238528 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="feast-init" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238533 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="feast-init" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238578 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8774cb9-f1d9-48a9-92db-12bfa7a4c2fe" containerName="postgres" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238585 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="registry" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238593 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a2b4bb-bf3d-40e9-982a-e0f090a46cd5" containerName="redis" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238599 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="online" Jun 08 11:58:00.238725 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.238624 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c74841-031f-4068-b784-227ac68d1c37" containerName="offline" Jun 08 11:58:00.241228 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.241213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" Jun 08 11:58:00.243806 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.243784 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jun 08 11:58:00.244561 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.244540 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-m7grh\"" Jun 08 11:58:00.244561 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.244553 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jun 08 11:58:00.258419 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.258343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-t7d6j"] Jun 08 11:58:00.288109 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.288059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:00.358265 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.358229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbh47\" (UniqueName: \"kubernetes.io/projected/0291ac6b-c417-42b4-b47f-059e52f28d54-kube-api-access-vbh47\") pod \"redis-69bbd798b6-t7d6j\" (UID: \"0291ac6b-c417-42b4-b47f-059e52f28d54\") " pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" Jun 08 11:58:00.459336 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.459305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbh47\" (UniqueName: \"kubernetes.io/projected/0291ac6b-c417-42b4-b47f-059e52f28d54-kube-api-access-vbh47\") pod \"redis-69bbd798b6-t7d6j\" (UID: \"0291ac6b-c417-42b4-b47f-059e52f28d54\") " pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" Jun 08 11:58:00.467777 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.467755 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbh47\" (UniqueName: \"kubernetes.io/projected/0291ac6b-c417-42b4-b47f-059e52f28d54-kube-api-access-vbh47\") pod \"redis-69bbd798b6-t7d6j\" (UID: \"0291ac6b-c417-42b4-b47f-059e52f28d54\") " pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" Jun 08 11:58:00.559753 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.559714 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" Jun 08 11:58:00.692767 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:00.692573 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-t7d6j"] Jun 08 11:58:00.695501 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:58:00.695472 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0291ac6b_c417_42b4_b47f_059e52f28d54.slice/crio-d080a382a42b4e2b0c1832f1e9392f16d2a12d281b7e9e134ba749cc24b0cd89 WatchSource:0}: Error finding container d080a382a42b4e2b0c1832f1e9392f16d2a12d281b7e9e134ba749cc24b0cd89: Status 404 returned error can't find the container with id d080a382a42b4e2b0c1832f1e9392f16d2a12d281b7e9e134ba749cc24b0cd89 Jun 08 11:58:01.666963 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:01.666929 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" event={"ID":"0291ac6b-c417-42b4-b47f-059e52f28d54","Type":"ContainerStarted","Data":"1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f"} Jun 08 11:58:01.667345 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:01.666972 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" event={"ID":"0291ac6b-c417-42b4-b47f-059e52f28d54","Type":"ContainerStarted","Data":"d080a382a42b4e2b0c1832f1e9392f16d2a12d281b7e9e134ba749cc24b0cd89"} Jun 08 11:58:01.683031 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:01.682983 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" podStartSLOduration=1.682968315 podStartE2EDuration="1.682968315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:58:01.682216169 +0000 UTC m=+1140.983992865" watchObservedRunningTime="2026-06-08 11:58:01.682968315 +0000 UTC m=+1140.984745031" Jun 08 11:58:05.689626 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.685058 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xsg4g"] Jun 08 11:58:05.693643 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.693591 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xsg4g"] Jun 08 11:58:05.693777 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.693716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:05.696047 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.696021 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jun 08 11:58:05.696574 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.696554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ed9a95e1-08eb-4837-b49e-60fb27981804-postgresdata\") pod \"postgres-864846d4f5-xsg4g\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:05.696688 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.696589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwqq\" (UniqueName: \"kubernetes.io/projected/ed9a95e1-08eb-4837-b49e-60fb27981804-kube-api-access-ptwqq\") pod \"postgres-864846d4f5-xsg4g\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:05.723980 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.723942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:05.797623 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.797585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ed9a95e1-08eb-4837-b49e-60fb27981804-postgresdata\") pod \"postgres-864846d4f5-xsg4g\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:05.797796 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.797641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwqq\" (UniqueName: \"kubernetes.io/projected/ed9a95e1-08eb-4837-b49e-60fb27981804-kube-api-access-ptwqq\") pod \"postgres-864846d4f5-xsg4g\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:05.798016 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.797996 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ed9a95e1-08eb-4837-b49e-60fb27981804-postgresdata\") pod \"postgres-864846d4f5-xsg4g\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:05.806325 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:05.806293 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwqq\" (UniqueName: \"kubernetes.io/projected/ed9a95e1-08eb-4837-b49e-60fb27981804-kube-api-access-ptwqq\") pod \"postgres-864846d4f5-xsg4g\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:06.005222 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:06.005125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 11:58:06.134716 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:06.134677 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xsg4g"] Jun 08 11:58:06.138880 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:58:06.138847 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded9a95e1_08eb_4837_b49e_60fb27981804.slice/crio-c2ca418524d97654141e2d60e50493a226351c073f0b10468c40ffd24ec56ae2 WatchSource:0}: Error finding container c2ca418524d97654141e2d60e50493a226351c073f0b10468c40ffd24ec56ae2: Status 404 returned error can't find the container with id c2ca418524d97654141e2d60e50493a226351c073f0b10468c40ffd24ec56ae2 Jun 08 11:58:06.693885 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:06.693842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" event={"ID":"ed9a95e1-08eb-4837-b49e-60fb27981804","Type":"ContainerStarted","Data":"d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4"} Jun 08 11:58:06.694412 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:06.693886 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" event={"ID":"ed9a95e1-08eb-4837-b49e-60fb27981804","Type":"ContainerStarted","Data":"c2ca418524d97654141e2d60e50493a226351c073f0b10468c40ffd24ec56ae2"} Jun 08 11:58:06.708310 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:06.708257 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" podStartSLOduration=1.708242481 podStartE2EDuration="1.708242481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:58:06.707775898 +0000 UTC m=+1146.009552609" watchObservedRunningTime="2026-06-08 11:58:06.708242481 +0000 UTC m=+1146.010019177" Jun 08 11:58:11.235410 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.235379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:11.327779 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.327746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:11.453891 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.453861 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc"] Jun 08 11:58:11.457132 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.457114 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.470835 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.470810 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jun 08 11:58:11.470835 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.470814 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jun 08 11:58:11.471032 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.470961 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-dzzgj\"" Jun 08 11:58:11.471235 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.471216 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jun 08 11:58:11.494854 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.494770 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc"] Jun 08 11:58:11.535491 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.535450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9wf\" (UniqueName: \"kubernetes.io/projected/a08f7552-f509-412d-8e76-82860fc0f071-kube-api-access-rz9wf\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.535705 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.535519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-registry-tls\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.535705 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.535557 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f7552-f509-412d-8e76-82860fc0f071-feast-data\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.535705 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.535582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.636293 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.636260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f7552-f509-412d-8e76-82860fc0f071-feast-data\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.636293 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.636299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.636530 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.636339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9wf\" (UniqueName: \"kubernetes.io/projected/a08f7552-f509-412d-8e76-82860fc0f071-kube-api-access-rz9wf\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.636530 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.636369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-registry-tls\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.636530 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:58:11.636471 2562 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Jun 08 11:58:11.636530 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:58:11.636531 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls podName:a08f7552-f509-412d-8e76-82860fc0f071 nodeName:}" failed. No retries permitted until 2026-06-08 11:58:12.136517212 +0000 UTC m=+1151.438293887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls") pod "feast-credit-scoring-b8f9968fc-bhhmc" (UID: "a08f7552-f509-412d-8e76-82860fc0f071") : secret "feast-credit-scoring-online-tls" not found Jun 08 11:58:11.636785 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.636661 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f7552-f509-412d-8e76-82860fc0f071-feast-data\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.639065 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.639039 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-registry-tls\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.650110 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.650084 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9wf\" (UniqueName: \"kubernetes.io/projected/a08f7552-f509-412d-8e76-82860fc0f071-kube-api-access-rz9wf\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:11.704837 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.704802 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:11.807993 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:11.807961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:12.142057 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:12.141957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:12.144499 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:12.144478 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls\") pod \"feast-credit-scoring-b8f9968fc-bhhmc\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:12.367969 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:12.367907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:12.503319 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:12.503287 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc"] Jun 08 11:58:12.506299 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:58:12.506269 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08f7552_f509_412d_8e76_82860fc0f071.slice/crio-2c3a049b0c685f32daa2fb80a2b2e1c837930935a379597fff39d781c06b1b1f WatchSource:0}: Error finding container 2c3a049b0c685f32daa2fb80a2b2e1c837930935a379597fff39d781c06b1b1f: Status 404 returned error can't find the container with id 2c3a049b0c685f32daa2fb80a2b2e1c837930935a379597fff39d781c06b1b1f Jun 08 11:58:12.719329 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:12.719239 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerStarted","Data":"a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178"} Jun 08 11:58:12.719329 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:12.719297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerStarted","Data":"2c3a049b0c685f32daa2fb80a2b2e1c837930935a379597fff39d781c06b1b1f"} Jun 08 11:58:13.725163 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:13.725067 2562 generic.go:356] "Generic (PLEG): container finished" podID="a08f7552-f509-412d-8e76-82860fc0f071" containerID="a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178" exitCode=0 Jun 08 11:58:13.725553 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:13.725149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerDied","Data":"a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178"} Jun 08 11:58:14.732004 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:14.731962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerStarted","Data":"573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50"} Jun 08 11:58:21.760917 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:21.760877 2562 generic.go:356] "Generic (PLEG): container finished" podID="a08f7552-f509-412d-8e76-82860fc0f071" containerID="573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50" exitCode=0 Jun 08 11:58:21.761308 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:21.760954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerDied","Data":"573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50"} Jun 08 11:58:22.768657 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:22.768590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerStarted","Data":"bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8"} Jun 08 11:58:22.769050 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:22.768693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerStarted","Data":"f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603"} Jun 08 11:58:22.791174 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:22.791115 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podStartSLOduration=11.791093552 podStartE2EDuration="11.791093552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:58:22.789945809 +0000 UTC m=+1162.091722505" watchObservedRunningTime="2026-06-08 11:58:22.791093552 +0000 UTC m=+1162.092870257" Jun 08 11:58:24.368352 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:24.368300 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:24.368871 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:24.368425 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:24.370009 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:24.369970 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jun 08 11:58:24.370160 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:24.370021 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 11:58:27.368409 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:27.368344 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jun 08 11:58:27.874605 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:27.874568 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:27.875007 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:27.874986 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:27.877838 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:27.877815 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:30.368685 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:30.368645 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jun 08 11:58:32.368154 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:32.368108 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:33.368844 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:33.368811 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:33.369478 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:33.369455 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 11:58:33.593868 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:33.593835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:37.058384 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.058323 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:37.160940 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.160905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:37.225777 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.225739 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f"] Jun 08 11:58:37.229326 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.229307 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.231422 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.231386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:37.231559 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.231434 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jun 08 11:58:37.231658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.231568 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jun 08 11:58:37.231658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.231572 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jun 08 11:58:37.231658 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.231654 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-lvflx\"" Jun 08 11:58:37.238859 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.238836 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f"] Jun 08 11:58:37.361442 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.361326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-registry-tls\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.361442 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.361379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0df57e60-aa48-4533-9863-472f0225999e-feast-data\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.361733 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.361455 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25qfj\" (UniqueName: \"kubernetes.io/projected/0df57e60-aa48-4533-9863-472f0225999e-kube-api-access-25qfj\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.361733 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.361523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.462392 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.462345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-registry-tls\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.462392 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.462396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0df57e60-aa48-4533-9863-472f0225999e-feast-data\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.462696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.462574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25qfj\" (UniqueName: \"kubernetes.io/projected/0df57e60-aa48-4533-9863-472f0225999e-kube-api-access-25qfj\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.462696 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.462662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.462817 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:58:37.462773 2562 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Jun 08 11:58:37.462817 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.462779 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0df57e60-aa48-4533-9863-472f0225999e-feast-data\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.462921 ip-10-0-137-95 kubenswrapper[2562]: E0608 11:58:37.462839 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls podName:0df57e60-aa48-4533-9863-472f0225999e nodeName:}" failed. No retries permitted until 2026-06-08 11:58:37.962819001 +0000 UTC m=+1177.264595676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls") pod "feast-driver-ranking-6f6747bccd-2lh6f" (UID: "0df57e60-aa48-4533-9863-472f0225999e") : secret "feast-driver-ranking-online-tls" not found Jun 08 11:58:37.465111 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.465086 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-registry-tls\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.470968 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.470946 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25qfj\" (UniqueName: \"kubernetes.io/projected/0df57e60-aa48-4533-9863-472f0225999e-kube-api-access-25qfj\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.829893 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.829851 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:37.967995 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.967951 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:37.970527 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:37.970503 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls\") pod \"feast-driver-ranking-6f6747bccd-2lh6f\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:38.140057 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:38.139952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:38.278158 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:38.278106 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f"] Jun 08 11:58:38.280298 ip-10-0-137-95 kubenswrapper[2562]: W0608 11:58:38.280263 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df57e60_aa48_4533_9863_472f0225999e.slice/crio-63dea4c5b5825e63b5bd843f8d7ab608bfeedc5288dbb4e81f9d48c869030faf WatchSource:0}: Error finding container 63dea4c5b5825e63b5bd843f8d7ab608bfeedc5288dbb4e81f9d48c869030faf: Status 404 returned error can't find the container with id 63dea4c5b5825e63b5bd843f8d7ab608bfeedc5288dbb4e81f9d48c869030faf Jun 08 11:58:38.840726 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:38.840686 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerStarted","Data":"88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27"} Jun 08 11:58:38.840926 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:38.840736 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerStarted","Data":"63dea4c5b5825e63b5bd843f8d7ab608bfeedc5288dbb4e81f9d48c869030faf"} Jun 08 11:58:43.864583 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:43.864548 2562 generic.go:356] "Generic (PLEG): container finished" podID="0df57e60-aa48-4533-9863-472f0225999e" containerID="88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27" exitCode=0 Jun 08 11:58:43.865109 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:43.864646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerDied","Data":"88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27"} Jun 08 11:58:44.870788 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:44.870758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerStarted","Data":"59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240"} Jun 08 11:58:47.926185 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:47.926151 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 11:58:50.899086 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:50.899050 2562 generic.go:356] "Generic (PLEG): container finished" podID="0df57e60-aa48-4533-9863-472f0225999e" containerID="59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240" exitCode=0 Jun 08 11:58:50.899438 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:50.899124 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerDied","Data":"59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240"} Jun 08 11:58:51.906374 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:51.906332 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerStarted","Data":"0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5"} Jun 08 11:58:51.906897 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:51.906449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerStarted","Data":"e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48"} Jun 08 11:58:51.930582 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:51.930524 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podStartSLOduration=14.930505791 podStartE2EDuration="14.930505791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 11:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 11:58:51.928943152 +0000 UTC m=+1191.230719863" watchObservedRunningTime="2026-06-08 11:58:51.930505791 +0000 UTC m=+1191.232282487" Jun 08 11:58:53.140398 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:53.140357 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:53.140795 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:53.140432 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:53.142081 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:53.142051 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 11:58:53.142081 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:53.142062 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Jun 08 11:58:56.141151 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:56.141103 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Jun 08 11:58:56.141661 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:56.141116 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 11:58:58.140557 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:58.140516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:58.140973 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:58.140579 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:59.141133 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:59.141086 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Jun 08 11:58:59.146639 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:59.146578 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:58:59.149627 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:58:59.149590 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:59:01.231448 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:59:01.231420 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:59:01.242649 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:59:01.242601 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 11:59:02.141108 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:59:02.141068 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 11:59:02.141876 ip-10-0-137-95 kubenswrapper[2562]: I0608 11:59:02.141859 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 12:00:15.858382 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:15.858297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:17.886789 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:17.886755 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:18.179544 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.179445 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:18.435342 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.435251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:18.470270 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.470230 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f"] Jun 08 12:00:18.470492 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.470465 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="registry" containerID="cri-o://e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48" gracePeriod=30 Jun 08 12:00:18.470567 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.470531 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" containerID="cri-o://0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5" gracePeriod=30 Jun 08 12:00:18.475051 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.475017 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc"] Jun 08 12:00:18.475327 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.475303 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" containerID="cri-o://bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8" gracePeriod=30 Jun 08 12:00:18.475406 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.475281 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="registry" containerID="cri-o://f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603" gracePeriod=30 Jun 08 12:00:18.501531 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:18.501501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:19.148081 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:19.148017 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 12:00:20.290458 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.290421 2562 generic.go:356] "Generic (PLEG): container finished" podID="0df57e60-aa48-4533-9863-472f0225999e" containerID="0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5" exitCode=0 Jun 08 12:00:20.290921 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.290490 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerDied","Data":"0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5"} Jun 08 12:00:20.292568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.292544 2562 generic.go:356] "Generic (PLEG): container finished" podID="a08f7552-f509-412d-8e76-82860fc0f071" containerID="bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8" exitCode=0 Jun 08 12:00:20.292705 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.292633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerDied","Data":"bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8"} Jun 08 12:00:20.835996 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.835953 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xsg4g"] Jun 08 12:00:20.836211 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.836158 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" podUID="ed9a95e1-08eb-4837-b49e-60fb27981804" containerName="postgres" containerID="cri-o://d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4" gracePeriod=30 Jun 08 12:00:20.844644 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.841244 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-t7d6j"] Jun 08 12:00:20.844644 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.841420 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" podUID="0291ac6b-c417-42b4-b47f-059e52f28d54" containerName="redis" containerID="cri-o://1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f" gracePeriod=30 Jun 08 12:00:20.891496 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:20.891458 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:21.084261 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.084235 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 12:00:21.120418 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.120390 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" Jun 08 12:00:21.165568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.165531 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptwqq\" (UniqueName: \"kubernetes.io/projected/ed9a95e1-08eb-4837-b49e-60fb27981804-kube-api-access-ptwqq\") pod \"ed9a95e1-08eb-4837-b49e-60fb27981804\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " Jun 08 12:00:21.165767 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.165596 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbh47\" (UniqueName: \"kubernetes.io/projected/0291ac6b-c417-42b4-b47f-059e52f28d54-kube-api-access-vbh47\") pod \"0291ac6b-c417-42b4-b47f-059e52f28d54\" (UID: \"0291ac6b-c417-42b4-b47f-059e52f28d54\") " Jun 08 12:00:21.165767 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.165684 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ed9a95e1-08eb-4837-b49e-60fb27981804-postgresdata\") pod \"ed9a95e1-08eb-4837-b49e-60fb27981804\" (UID: \"ed9a95e1-08eb-4837-b49e-60fb27981804\") " Jun 08 12:00:21.167912 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.167878 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9a95e1-08eb-4837-b49e-60fb27981804-kube-api-access-ptwqq" (OuterVolumeSpecName: "kube-api-access-ptwqq") pod "ed9a95e1-08eb-4837-b49e-60fb27981804" (UID: "ed9a95e1-08eb-4837-b49e-60fb27981804"). InnerVolumeSpecName "kube-api-access-ptwqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:00:21.168024 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.167937 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0291ac6b-c417-42b4-b47f-059e52f28d54-kube-api-access-vbh47" (OuterVolumeSpecName: "kube-api-access-vbh47") pod "0291ac6b-c417-42b4-b47f-059e52f28d54" (UID: "0291ac6b-c417-42b4-b47f-059e52f28d54"). InnerVolumeSpecName "kube-api-access-vbh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:00:21.195498 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.195456 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a95e1-08eb-4837-b49e-60fb27981804-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "ed9a95e1-08eb-4837-b49e-60fb27981804" (UID: "ed9a95e1-08eb-4837-b49e-60fb27981804"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:00:21.266355 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.266309 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwqq\" (UniqueName: \"kubernetes.io/projected/ed9a95e1-08eb-4837-b49e-60fb27981804-kube-api-access-ptwqq\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:21.266355 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.266348 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbh47\" (UniqueName: \"kubernetes.io/projected/0291ac6b-c417-42b4-b47f-059e52f28d54-kube-api-access-vbh47\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:21.266355 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.266359 2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ed9a95e1-08eb-4837-b49e-60fb27981804-postgresdata\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:21.297916 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.297881 2562 generic.go:356] "Generic (PLEG): container finished" podID="0291ac6b-c417-42b4-b47f-059e52f28d54" containerID="1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f" exitCode=0 Jun 08 12:00:21.298367 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.297939 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" Jun 08 12:00:21.298367 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.297969 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" event={"ID":"0291ac6b-c417-42b4-b47f-059e52f28d54","Type":"ContainerDied","Data":"1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f"} Jun 08 12:00:21.298367 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.298009 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-t7d6j" event={"ID":"0291ac6b-c417-42b4-b47f-059e52f28d54","Type":"ContainerDied","Data":"d080a382a42b4e2b0c1832f1e9392f16d2a12d281b7e9e134ba749cc24b0cd89"} Jun 08 12:00:21.298367 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.298031 2562 scope.go:117] "RemoveContainer" containerID="1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f" Jun 08 12:00:21.299369 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.299341 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed9a95e1-08eb-4837-b49e-60fb27981804" containerID="d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4" exitCode=0 Jun 08 12:00:21.299480 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.299369 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" event={"ID":"ed9a95e1-08eb-4837-b49e-60fb27981804","Type":"ContainerDied","Data":"d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4"} Jun 08 12:00:21.299480 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.299408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" event={"ID":"ed9a95e1-08eb-4837-b49e-60fb27981804","Type":"ContainerDied","Data":"c2ca418524d97654141e2d60e50493a226351c073f0b10468c40ffd24ec56ae2"} Jun 08 12:00:21.299480 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.299416 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-xsg4g" Jun 08 12:00:21.306579 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.306562 2562 scope.go:117] "RemoveContainer" containerID="1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f" Jun 08 12:00:21.306900 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:21.306877 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f\": container with ID starting with 1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f not found: ID does not exist" containerID="1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f" Jun 08 12:00:21.307000 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.306908 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f"} err="failed to get container status \"1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f\": rpc error: code = NotFound desc = could not find container \"1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f\": container with ID starting with 1ce981d8b703fb56e9f35c8ffda227afec3214ec8d1c2a8cd4c28b6e6966988f not found: ID does not exist" Jun 08 12:00:21.307000 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.306925 2562 scope.go:117] "RemoveContainer" containerID="d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4" Jun 08 12:00:21.314553 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.314533 2562 scope.go:117] "RemoveContainer" containerID="d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4" Jun 08 12:00:21.314916 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:21.314888 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4\": container with ID starting with d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4 not found: ID does not exist" containerID="d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4" Jun 08 12:00:21.315012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.314920 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4"} err="failed to get container status \"d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4\": rpc error: code = NotFound desc = could not find container \"d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4\": container with ID starting with d148636497938c5472a8f37bb75be55b9263a94cbcc685518818ffd0096ee2c4 not found: ID does not exist" Jun 08 12:00:21.316142 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.316121 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-t7d6j"] Jun 08 12:00:21.319476 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.319452 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-t7d6j"] Jun 08 12:00:21.328679 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.328657 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xsg4g"] Jun 08 12:00:21.330323 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:21.330305 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-xsg4g"] Jun 08 12:00:23.202282 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:23.202249 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0291ac6b-c417-42b4-b47f-059e52f28d54" path="/var/lib/kubelet/pods/0291ac6b-c417-42b4-b47f-059e52f28d54/volumes" Jun 08 12:00:23.202696 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:23.202601 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed9a95e1-08eb-4837-b49e-60fb27981804" path="/var/lib/kubelet/pods/ed9a95e1-08eb-4837-b49e-60fb27981804/volumes" Jun 08 12:00:23.332092 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:23.332056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:23.404210 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:23.404175 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:27.876287 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:27.876241 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 12:00:28.381893 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:28.381852 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:28.479828 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:28.479726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:28.919763 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:28.919706 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:29.148037 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:29.147992 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 12:00:37.875932 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:37.875888 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 12:00:39.147954 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:39.147902 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Jun 08 12:00:39.148351 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:39.148031 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 12:00:45.082068 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:45.082030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:45.180161 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:45.180131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:47.875848 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:47.875795 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jun 08 12:00:47.876453 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:47.875942 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 12:00:48.675960 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.675931 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 12:00:48.714576 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.714534 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25qfj\" (UniqueName: \"kubernetes.io/projected/0df57e60-aa48-4533-9863-472f0225999e-kube-api-access-25qfj\") pod \"0df57e60-aa48-4533-9863-472f0225999e\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " Jun 08 12:00:48.714793 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.714595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls\") pod \"0df57e60-aa48-4533-9863-472f0225999e\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " Jun 08 12:00:48.714793 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.714647 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0df57e60-aa48-4533-9863-472f0225999e-feast-data\") pod \"0df57e60-aa48-4533-9863-472f0225999e\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " Jun 08 12:00:48.714793 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.714689 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-registry-tls\") pod \"0df57e60-aa48-4533-9863-472f0225999e\" (UID: \"0df57e60-aa48-4533-9863-472f0225999e\") " Jun 08 12:00:48.715151 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.715121 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df57e60-aa48-4533-9863-472f0225999e-feast-data" (OuterVolumeSpecName: "feast-data") pod "0df57e60-aa48-4533-9863-472f0225999e" (UID: "0df57e60-aa48-4533-9863-472f0225999e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:00:48.716920 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.716887 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls" (OuterVolumeSpecName: "online-tls") pod "0df57e60-aa48-4533-9863-472f0225999e" (UID: "0df57e60-aa48-4533-9863-472f0225999e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:00:48.717037 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.716943 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df57e60-aa48-4533-9863-472f0225999e-kube-api-access-25qfj" (OuterVolumeSpecName: "kube-api-access-25qfj") pod "0df57e60-aa48-4533-9863-472f0225999e" (UID: "0df57e60-aa48-4533-9863-472f0225999e"). InnerVolumeSpecName "kube-api-access-25qfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:00:48.717037 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.716954 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0df57e60-aa48-4533-9863-472f0225999e" (UID: "0df57e60-aa48-4533-9863-472f0225999e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:00:48.815761 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.815718 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:48.815761 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.815758 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25qfj\" (UniqueName: \"kubernetes.io/projected/0df57e60-aa48-4533-9863-472f0225999e-kube-api-access-25qfj\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:48.815977 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.815774 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0df57e60-aa48-4533-9863-472f0225999e-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:48.815977 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:48.815787 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0df57e60-aa48-4533-9863-472f0225999e-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:49.129400 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.129371 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 12:00:49.219547 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.219513 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls\") pod \"a08f7552-f509-412d-8e76-82860fc0f071\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " Jun 08 12:00:49.219547 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.219561 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz9wf\" (UniqueName: \"kubernetes.io/projected/a08f7552-f509-412d-8e76-82860fc0f071-kube-api-access-rz9wf\") pod \"a08f7552-f509-412d-8e76-82860fc0f071\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " Jun 08 12:00:49.219818 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.219590 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-registry-tls\") pod \"a08f7552-f509-412d-8e76-82860fc0f071\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " Jun 08 12:00:49.219818 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.219707 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f7552-f509-412d-8e76-82860fc0f071-feast-data\") pod \"a08f7552-f509-412d-8e76-82860fc0f071\" (UID: \"a08f7552-f509-412d-8e76-82860fc0f071\") " Jun 08 12:00:49.229442 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.229395 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08f7552-f509-412d-8e76-82860fc0f071-feast-data" (OuterVolumeSpecName: "feast-data") pod "a08f7552-f509-412d-8e76-82860fc0f071" (UID: "a08f7552-f509-412d-8e76-82860fc0f071"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:00:49.231985 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.231961 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a08f7552-f509-412d-8e76-82860fc0f071" (UID: "a08f7552-f509-412d-8e76-82860fc0f071"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:00:49.246132 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.246102 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls" (OuterVolumeSpecName: "online-tls") pod "a08f7552-f509-412d-8e76-82860fc0f071" (UID: "a08f7552-f509-412d-8e76-82860fc0f071"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:00:49.246253 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.246102 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08f7552-f509-412d-8e76-82860fc0f071-kube-api-access-rz9wf" (OuterVolumeSpecName: "kube-api-access-rz9wf") pod "a08f7552-f509-412d-8e76-82860fc0f071" (UID: "a08f7552-f509-412d-8e76-82860fc0f071"). InnerVolumeSpecName "kube-api-access-rz9wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:00:49.320656 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.320590 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a08f7552-f509-412d-8e76-82860fc0f071-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:49.320656 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.320652 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:49.320835 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.320663 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9wf\" (UniqueName: \"kubernetes.io/projected/a08f7552-f509-412d-8e76-82860fc0f071-kube-api-access-rz9wf\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:49.320835 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.320673 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a08f7552-f509-412d-8e76-82860fc0f071-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:00:49.413407 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.413308 2562 generic.go:356] "Generic (PLEG): container finished" podID="a08f7552-f509-412d-8e76-82860fc0f071" containerID="f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603" exitCode=137 Jun 08 12:00:49.413407 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.413363 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" Jun 08 12:00:49.413407 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.413393 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerDied","Data":"f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603"} Jun 08 12:00:49.413713 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.413454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc" event={"ID":"a08f7552-f509-412d-8e76-82860fc0f071","Type":"ContainerDied","Data":"2c3a049b0c685f32daa2fb80a2b2e1c837930935a379597fff39d781c06b1b1f"} Jun 08 12:00:49.413713 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.413482 2562 scope.go:117] "RemoveContainer" containerID="bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8" Jun 08 12:00:49.416001 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.415968 2562 generic.go:356] "Generic (PLEG): container finished" podID="0df57e60-aa48-4533-9863-472f0225999e" containerID="e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48" exitCode=137 Jun 08 12:00:49.416114 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.416024 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerDied","Data":"e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48"} Jun 08 12:00:49.416114 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.416031 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" Jun 08 12:00:49.416114 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.416069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f" event={"ID":"0df57e60-aa48-4533-9863-472f0225999e","Type":"ContainerDied","Data":"63dea4c5b5825e63b5bd843f8d7ab608bfeedc5288dbb4e81f9d48c869030faf"} Jun 08 12:00:49.423594 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.423575 2562 scope.go:117] "RemoveContainer" containerID="f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603" Jun 08 12:00:49.431986 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.431958 2562 scope.go:117] "RemoveContainer" containerID="573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50" Jun 08 12:00:49.440234 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.440211 2562 scope.go:117] "RemoveContainer" containerID="a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178" Jun 08 12:00:49.448816 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.448790 2562 scope.go:117] "RemoveContainer" containerID="bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8" Jun 08 12:00:49.449380 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.449263 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8\": container with ID starting with bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8 not found: ID does not exist" containerID="bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8" Jun 08 12:00:49.449380 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.449306 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8"} err="failed to get container status \"bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8\": rpc error: code = NotFound desc = could not find container \"bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8\": container with ID starting with bd7191445d0cec767681fdc81bca38b08293e55af1e7f303d2c12d5bb442e9d8 not found: ID does not exist" Jun 08 12:00:49.449380 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.449337 2562 scope.go:117] "RemoveContainer" containerID="f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603" Jun 08 12:00:49.449952 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.449929 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603\": container with ID starting with f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603 not found: ID does not exist" containerID="f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603" Jun 08 12:00:49.450146 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.449958 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603"} err="failed to get container status \"f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603\": rpc error: code = NotFound desc = could not find container \"f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603\": container with ID starting with f197c46abee1cf0c09a40a3413ae734ac46de08c4553305286a37447c212c603 not found: ID does not exist" Jun 08 12:00:49.450146 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.449976 2562 scope.go:117] "RemoveContainer" containerID="573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50" Jun 08 12:00:49.450274 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.450261 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50\": container with ID starting with 573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50 not found: ID does not exist" containerID="573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50" Jun 08 12:00:49.450335 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.450279 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50"} err="failed to get container status \"573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50\": rpc error: code = NotFound desc = could not find container \"573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50\": container with ID starting with 573912774de53abe8f303e507a036da1e595347d8f432fa9b315020f12ffbb50 not found: ID does not exist" Jun 08 12:00:49.450335 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.450291 2562 scope.go:117] "RemoveContainer" containerID="a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178" Jun 08 12:00:49.450571 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.450550 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178\": container with ID starting with a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178 not found: ID does not exist" containerID="a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178" Jun 08 12:00:49.450658 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.450576 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178"} err="failed to get container status \"a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178\": rpc error: code = NotFound desc = could not find container \"a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178\": container with ID starting with a9c4695f3f13c3a2c947743b0994431f2f65a867f457edf5cadb50fb80bdf178 not found: ID does not exist" Jun 08 12:00:49.450658 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.450592 2562 scope.go:117] "RemoveContainer" containerID="0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5" Jun 08 12:00:49.451496 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.451476 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f"] Jun 08 12:00:49.457696 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.457665 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6f6747bccd-2lh6f"] Jun 08 12:00:49.460081 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.460060 2562 scope.go:117] "RemoveContainer" containerID="e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48" Jun 08 12:00:49.468766 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.468744 2562 scope.go:117] "RemoveContainer" containerID="59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240" Jun 08 12:00:49.471627 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.471591 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc"] Jun 08 12:00:49.477306 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.477178 2562 scope.go:117] "RemoveContainer" containerID="88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27" Jun 08 12:00:49.482981 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.482955 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-b8f9968fc-bhhmc"] Jun 08 12:00:49.486810 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.486787 2562 scope.go:117] "RemoveContainer" containerID="0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5" Jun 08 12:00:49.487127 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.487109 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5\": container with ID starting with 0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5 not found: ID does not exist" containerID="0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5" Jun 08 12:00:49.487192 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.487135 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5"} err="failed to get container status \"0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5\": rpc error: code = NotFound desc = could not find container \"0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5\": container with ID starting with 0cf116c38457c8f09d1fae40797687807e0f4d12fcae7b610b3680e36bc4faa5 not found: ID does not exist" Jun 08 12:00:49.487192 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.487153 2562 scope.go:117] "RemoveContainer" containerID="e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48" Jun 08 12:00:49.487440 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.487418 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48\": container with ID starting with e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48 not found: ID does not exist" containerID="e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48" Jun 08 12:00:49.487504 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.487459 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48"} err="failed to get container status \"e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48\": rpc error: code = NotFound desc = could not find container \"e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48\": container with ID starting with e2710c02de62e8dd06a0f13a2fa34b0aaa34273c21314e959e91766db4dc9d48 not found: ID does not exist" Jun 08 12:00:49.487504 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.487485 2562 scope.go:117] "RemoveContainer" containerID="59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240" Jun 08 12:00:49.487897 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.487772 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240\": container with ID starting with 59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240 not found: ID does not exist" containerID="59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240" Jun 08 12:00:49.487897 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.487799 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240"} err="failed to get container status \"59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240\": rpc error: code = NotFound desc = could not find container \"59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240\": container with ID starting with 59fe95c0b94eae9c192ec0154606afef4f45a48791016d4cc3df284fdf3de240 not found: ID does not exist" Jun 08 12:00:49.487897 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.487813 2562 scope.go:117] "RemoveContainer" containerID="88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27" Jun 08 12:00:49.488159 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:49.488036 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27\": container with ID starting with 88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27 not found: ID does not exist" containerID="88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27" Jun 08 12:00:49.488159 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:49.488057 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27"} err="failed to get container status \"88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27\": rpc error: code = NotFound desc = could not find container \"88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27\": container with ID starting with 88bbf3c5243dc5d236056b67d58e358f154a6c1bee117ad9d780134274e58d27 not found: ID does not exist" Jun 08 12:00:51.202465 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:51.202432 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0df57e60-aa48-4533-9863-472f0225999e" path="/var/lib/kubelet/pods/0df57e60-aa48-4533-9863-472f0225999e/volumes" Jun 08 12:00:51.202977 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:51.202962 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a08f7552-f509-412d-8e76-82860fc0f071" path="/var/lib/kubelet/pods/a08f7552-f509-412d-8e76-82860fc0f071/volumes" Jun 08 12:00:55.853839 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.853806 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:55.941703 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.941671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:55.946538 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946507 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7"] Jun 08 12:00:55.946828 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946815 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="registry" Jun 08 12:00:55.946875 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946830 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="registry" Jun 08 12:00:55.946875 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946843 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0291ac6b-c417-42b4-b47f-059e52f28d54" containerName="redis" Jun 08 12:00:55.946875 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946850 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0291ac6b-c417-42b4-b47f-059e52f28d54" containerName="redis" Jun 08 12:00:55.946875 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946862 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="registry" Jun 08 12:00:55.946875 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946867 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="registry" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946877 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed9a95e1-08eb-4837-b49e-60fb27981804" containerName="postgres" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946882 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed9a95e1-08eb-4837-b49e-60fb27981804" containerName="postgres" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946891 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="feast-apply" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946897 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="feast-apply" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946904 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="feast-apply" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946908 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="feast-apply" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946915 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946919 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946924 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="feast-init" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946929 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="feast-init" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946935 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946939 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946947 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="feast-init" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946952 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="feast-init" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.946993 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="online" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.947000 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0291ac6b-c417-42b4-b47f-059e52f28d54" containerName="redis" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.947009 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="registry" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.947015 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed9a95e1-08eb-4837-b49e-60fb27981804" containerName="postgres" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.947021 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a08f7552-f509-412d-8e76-82860fc0f071" containerName="online" Jun 08 12:00:55.947020 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.947028 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df57e60-aa48-4533-9863-472f0225999e" containerName="registry" Jun 08 12:00:55.950133 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.950116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:55.953785 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.953759 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 08 12:00:55.954060 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.954040 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 12:00:55.954142 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.954124 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 08 12:00:55.954286 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.954271 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 08 12:00:55.954363 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.954350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nbmd2\"" Jun 08 12:00:55.954745 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.954730 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 12:00:55.954846 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.954829 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 08 12:00:55.962682 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.962653 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7"] Jun 08 12:00:55.971720 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.971679 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:55.971720 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.971722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:55.971969 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.971752 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:55.971969 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.971776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:55.971969 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.971813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b573d032-6595-49e5-87e8-a6b2b2b2829a-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:55.971969 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:55.971879 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/b573d032-6595-49e5-87e8-a6b2b2b2829a-kube-api-access-8j66l\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.073029 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.072974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/b573d032-6595-49e5-87e8-a6b2b2b2829a-kube-api-access-8j66l\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.073029 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.073037 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.073076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.073114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.073138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.073190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b573d032-6595-49e5-87e8-a6b2b2b2829a-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073214 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073222 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073267 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073213 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073289 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls podName:b573d032-6595-49e5-87e8-a6b2b2b2829a nodeName:}" failed. No retries permitted until 2026-06-08 12:00:56.573272411 +0000 UTC m=+1315.875049085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls") pod "feast-simple-feast-setup-75c64c888b-95rj7" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073309 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls podName:b573d032-6595-49e5-87e8-a6b2b2b2829a nodeName:}" failed. No retries permitted until 2026-06-08 12:00:56.573296374 +0000 UTC m=+1315.875073049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls") pod "feast-simple-feast-setup-75c64c888b-95rj7" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a") : secret "feast-simple-feast-setup-offline-tls" not found Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073321 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls podName:b573d032-6595-49e5-87e8-a6b2b2b2829a nodeName:}" failed. No retries permitted until 2026-06-08 12:00:56.573315848 +0000 UTC m=+1315.875092523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls") pod "feast-simple-feast-setup-75c64c888b-95rj7" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a") : secret "feast-simple-feast-setup-online-tls" not found Jun 08 12:00:56.073317 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.073330 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls podName:b573d032-6595-49e5-87e8-a6b2b2b2829a nodeName:}" failed. No retries permitted until 2026-06-08 12:00:56.573325088 +0000 UTC m=+1315.875101763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls") pod "feast-simple-feast-setup-75c64c888b-95rj7" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a") : secret "feast-simple-feast-setup-ui-tls" not found Jun 08 12:00:56.073841 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.073542 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b573d032-6595-49e5-87e8-a6b2b2b2829a-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.081941 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.081919 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/b573d032-6595-49e5-87e8-a6b2b2b2829a-kube-api-access-8j66l\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.132011 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.131930 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:56.227543 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.227504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:56.578440 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.578377 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.578440 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.578446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.578710 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.578475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.578710 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.578498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.578710 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.578535 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:00:56.578710 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:00:56.578638 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls podName:b573d032-6595-49e5-87e8-a6b2b2b2829a nodeName:}" failed. No retries permitted until 2026-06-08 12:00:57.578596235 +0000 UTC m=+1316.880372913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls") pod "feast-simple-feast-setup-75c64c888b-95rj7" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:00:56.581187 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.581161 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.581407 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.581385 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.581451 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.581385 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:56.601042 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.601010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:56.744373 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:56.744335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:57.586450 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:57.586395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:57.588995 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:57.588969 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-95rj7\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:57.766692 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:57.766649 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:00:57.909642 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:57.909585 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7"] Jun 08 12:00:57.911949 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:00:57.911920 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb573d032_6595_49e5_87e8_a6b2b2b2829a.slice/crio-dd611067e84a3243a420fe81a12a4d926858811f763cc7ac7ef8a32cd5300ed2 WatchSource:0}: Error finding container dd611067e84a3243a420fe81a12a4d926858811f763cc7ac7ef8a32cd5300ed2: Status 404 returned error can't find the container with id dd611067e84a3243a420fe81a12a4d926858811f763cc7ac7ef8a32cd5300ed2 Jun 08 12:00:57.913786 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:57.913768 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 12:00:58.455682 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:58.455645 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerStarted","Data":"dd611067e84a3243a420fe81a12a4d926858811f763cc7ac7ef8a32cd5300ed2"} Jun 08 12:00:58.664818 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:58.664717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:58.765220 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:58.765028 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:00:58.866096 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:00:58.865961 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:01:12.524219 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:12.524176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerStarted","Data":"f3141a0ccf498fdaae1f71adeeddba03c64c7552d50d7ead23524d73412913b2"} Jun 08 12:01:17.552196 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:17.552156 2562 generic.go:356] "Generic (PLEG): container finished" podID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerID="f3141a0ccf498fdaae1f71adeeddba03c64c7552d50d7ead23524d73412913b2" exitCode=0 Jun 08 12:01:17.552196 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:17.552176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerDied","Data":"f3141a0ccf498fdaae1f71adeeddba03c64c7552d50d7ead23524d73412913b2"} Jun 08 12:01:18.558655 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:18.558593 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerStarted","Data":"dca8461a82f852fc15fd4f764760b1b0c8f24d520f5c19374bbaf1660cbf256e"} Jun 08 12:01:23.579067 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:23.579031 2562 generic.go:356] "Generic (PLEG): container finished" podID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerID="dca8461a82f852fc15fd4f764760b1b0c8f24d520f5c19374bbaf1660cbf256e" exitCode=0 Jun 08 12:01:23.579495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:23.579100 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerDied","Data":"dca8461a82f852fc15fd4f764760b1b0c8f24d520f5c19374bbaf1660cbf256e"} Jun 08 12:01:24.589676 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.589633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerStarted","Data":"200dffe55e217af969877d95569c858f987e5bba840fc42329a335a525701084"} Jun 08 12:01:24.590177 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.589703 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerStarted","Data":"7867da134927ed41114f4b6ff71729ab218e669c5c679a62739c8fdbecd1276b"} Jun 08 12:01:24.590177 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.589725 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerStarted","Data":"73152133f20e46a127acf99d5cf5b3bff8e21ee1686633de3d8355b99fa3eaca"} Jun 08 12:01:24.590177 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.589749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerStarted","Data":"8051e982608d4b2780be05bec5d9395d0f6e812dcda5e00b81600416c97033b7"} Jun 08 12:01:24.615061 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.614983 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podStartSLOduration=15.510740194 podStartE2EDuration="29.61495757s" totalImagesPullingTime="14.104217376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 12:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:01:24.612853174 +0000 UTC m=+1343.914629871" watchObservedRunningTime="2026-06-08 12:01:24.61495757 +0000 UTC m=+1343.916734268" Jun 08 12:01:24.767551 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.767431 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:24.767551 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.767498 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:24.767551 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.767518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:24.768055 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.768025 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:24.769736 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.769537 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Jun 08 12:01:24.769736 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.769560 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 12:01:24.769736 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.769592 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 12:01:24.769736 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:24.769660 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Jun 08 12:01:27.767052 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767004 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:27.767545 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767155 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:27.767545 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767164 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 12:01:27.767545 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767222 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused" Jun 08 12:01:27.767793 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767661 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:27.767793 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767695 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:27.767793 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767739 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 12:01:27.767949 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:27.767915 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Jun 08 12:01:30.768819 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:30.768788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:30.769211 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:30.768900 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:30.769211 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:30.769119 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:30.769659 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:30.769594 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:30.769772 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:30.769727 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:30.769816 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:30.769784 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:30.773271 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:30.773254 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:01:31.630344 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:01:31.630310 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:02:18.249270 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:18.249233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:18.703750 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:18.703716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:27.925457 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:27.925392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:34.649669 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.649230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:34.652315 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.652285 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7"] Jun 08 12:02:34.652516 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.652493 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="registry" containerID="cri-o://8051e982608d4b2780be05bec5d9395d0f6e812dcda5e00b81600416c97033b7" gracePeriod=30 Jun 08 12:02:34.652883 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.652855 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" containerID="cri-o://200dffe55e217af969877d95569c858f987e5bba840fc42329a335a525701084" gracePeriod=30 Jun 08 12:02:34.653435 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.653373 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" containerID="cri-o://73152133f20e46a127acf99d5cf5b3bff8e21ee1686633de3d8355b99fa3eaca" gracePeriod=30 Jun 08 12:02:34.653556 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.653493 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="offline" containerID="cri-o://7867da134927ed41114f4b6ff71729ab218e669c5c679a62739c8fdbecd1276b" gracePeriod=30 Jun 08 12:02:34.745091 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.745062 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:34.845723 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.845692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:34.973806 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.973713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:34.981692 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.981665 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28"] Jun 08 12:02:34.985277 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.985261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:34.988722 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:34.988698 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bn4zd\"" Jun 08 12:02:35.001295 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.001266 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28"] Jun 08 12:02:35.039124 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.039088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.039327 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.039133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjfxb\" (UniqueName: \"kubernetes.io/projected/bed66c42-16bd-4f78-8a3e-a694f44b606e-kube-api-access-hjfxb\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.039327 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.039159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.039327 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.039223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bed66c42-16bd-4f78-8a3e-a694f44b606e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.039327 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.039264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.039327 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.039301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.070687 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.070642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:35.140684 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.140653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.140847 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.140702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.140847 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.140833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjfxb\" (UniqueName: \"kubernetes.io/projected/bed66c42-16bd-4f78-8a3e-a694f44b606e-kube-api-access-hjfxb\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.140961 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.140866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.140961 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.140910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bed66c42-16bd-4f78-8a3e-a694f44b606e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.140961 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.140946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.141100 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:02:35.141055 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jun 08 12:02:35.141100 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:02:35.141080 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 08 12:02:35.141265 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:02:35.141132 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls podName:bed66c42-16bd-4f78-8a3e-a694f44b606e nodeName:}" failed. No retries permitted until 2026-06-08 12:02:35.641110589 +0000 UTC m=+1414.942887279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls") pod "feast-simple-feast-setup-75c64c888b-zsk28" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e") : secret "feast-simple-feast-setup-ui-tls" not found Jun 08 12:02:35.141265 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:02:35.141149 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls podName:bed66c42-16bd-4f78-8a3e-a694f44b606e nodeName:}" failed. No retries permitted until 2026-06-08 12:02:35.641140956 +0000 UTC m=+1414.942917630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls") pod "feast-simple-feast-setup-75c64c888b-zsk28" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e") : secret "feast-simple-feast-setup-online-tls" not found Jun 08 12:02:35.141381 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.141323 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bed66c42-16bd-4f78-8a3e-a694f44b606e-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.143354 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.143329 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.143812 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.143795 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.150219 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.150186 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjfxb\" (UniqueName: \"kubernetes.io/projected/bed66c42-16bd-4f78-8a3e-a694f44b606e-kube-api-access-hjfxb\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.169423 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.169388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:35.271271 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.271180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:35.444645 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.444589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:02:35.644282 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.644233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.644458 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.644329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.646747 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.646717 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.646861 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.646822 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-zsk28\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:35.880433 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.880398 2562 generic.go:356] "Generic (PLEG): container finished" podID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerID="200dffe55e217af969877d95569c858f987e5bba840fc42329a335a525701084" exitCode=0 Jun 08 12:02:35.880433 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.880426 2562 generic.go:356] "Generic (PLEG): container finished" podID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerID="73152133f20e46a127acf99d5cf5b3bff8e21ee1686633de3d8355b99fa3eaca" exitCode=0 Jun 08 12:02:35.880858 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.880477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerDied","Data":"200dffe55e217af969877d95569c858f987e5bba840fc42329a335a525701084"} Jun 08 12:02:35.880858 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.880519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerDied","Data":"73152133f20e46a127acf99d5cf5b3bff8e21ee1686633de3d8355b99fa3eaca"} Jun 08 12:02:35.894740 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:35.894664 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:36.031575 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:36.031545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28"] Jun 08 12:02:36.033349 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:02:36.033325 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbed66c42_16bd_4f78_8a3e_a694f44b606e.slice/crio-91200fb89f2875a3012eec54287766c4b7d5c027756b478d9d07c2a7fda466e3 WatchSource:0}: Error finding container 91200fb89f2875a3012eec54287766c4b7d5c027756b478d9d07c2a7fda466e3: Status 404 returned error can't find the container with id 91200fb89f2875a3012eec54287766c4b7d5c027756b478d9d07c2a7fda466e3 Jun 08 12:02:36.888259 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:36.888218 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerStarted","Data":"502169d466256fe669d4ca86354e85451c23e3c07582cdba06c204d9d1bc0c9f"} Jun 08 12:02:36.888753 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:36.888271 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerStarted","Data":"91200fb89f2875a3012eec54287766c4b7d5c027756b478d9d07c2a7fda466e3"} Jun 08 12:02:40.770330 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:40.770279 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 12:02:40.906933 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:40.906896 2562 generic.go:356] "Generic (PLEG): container finished" podID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerID="502169d466256fe669d4ca86354e85451c23e3c07582cdba06c204d9d1bc0c9f" exitCode=0 Jun 08 12:02:40.907118 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:40.906969 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerDied","Data":"502169d466256fe669d4ca86354e85451c23e3c07582cdba06c204d9d1bc0c9f"} Jun 08 12:02:41.627661 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:41.627579 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 12:02:41.913673 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:41.913557 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerStarted","Data":"9c8aed27c7b2541ad447790360c8614277d25f45ef9ebe4cef82b0346722a836"} Jun 08 12:02:46.946065 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:46.945972 2562 generic.go:356] "Generic (PLEG): container finished" podID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerID="9c8aed27c7b2541ad447790360c8614277d25f45ef9ebe4cef82b0346722a836" exitCode=0 Jun 08 12:02:46.946065 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:46.946029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerDied","Data":"9c8aed27c7b2541ad447790360c8614277d25f45ef9ebe4cef82b0346722a836"} Jun 08 12:02:47.956120 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:47.956081 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerStarted","Data":"89d6042c6a2adb89c0da1dd2a8d4d8aa0fcab39bf038ffddb65390bab7ad743c"} Jun 08 12:02:47.956633 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:47.956144 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerStarted","Data":"19699d9723ebf38700f320abfc2e23bd305790abf3dd23da7a6877cf1f412690"} Jun 08 12:02:47.956633 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:47.956166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerStarted","Data":"231a68779f722406f6e19d8c9b1967888b06303ddfad91b84d41d218d806d34b"} Jun 08 12:02:47.956633 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:47.956184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerStarted","Data":"99a4ad616ad47b6bbcedf762447488803742ebaafce6121a876c68a7176004f1"} Jun 08 12:02:47.981183 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:47.981108 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podStartSLOduration=13.981083109 podStartE2EDuration="13.981083109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 12:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:02:47.980117774 +0000 UTC m=+1427.281894472" watchObservedRunningTime="2026-06-08 12:02:47.981083109 +0000 UTC m=+1427.282859806" Jun 08 12:02:50.770181 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.770081 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 12:02:50.895886 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.895730 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:50.895886 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.895792 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:50.895886 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.895813 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:50.895886 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.895833 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:50.897897 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.897853 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 12:02:50.897897 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.897884 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused" Jun 08 12:02:50.898123 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.897884 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused" Jun 08 12:02:50.898123 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:50.897940 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 12:02:51.627592 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:51.627544 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 12:02:53.896248 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.896215 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.896657 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.896457 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.896657 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.896589 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.896973 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.896948 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.897087 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.896990 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.897087 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.897009 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.897087 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.897074 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.897388 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.897368 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.897476 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.897434 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.900853 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.900832 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.987518 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.987492 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:02:53.990343 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:02:53.990328 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:03:00.769770 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:00.769722 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused" Jun 08 12:03:00.770233 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:00.769860 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:03:01.628127 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:01.628078 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Jun 08 12:03:01.628314 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:01.628201 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:03:05.033079 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.033041 2562 generic.go:356] "Generic (PLEG): container finished" podID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerID="7867da134927ed41114f4b6ff71729ab218e669c5c679a62739c8fdbecd1276b" exitCode=137 Jun 08 12:03:05.033079 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.033071 2562 generic.go:356] "Generic (PLEG): container finished" podID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerID="8051e982608d4b2780be05bec5d9395d0f6e812dcda5e00b81600416c97033b7" exitCode=137 Jun 08 12:03:05.033483 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.033091 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerDied","Data":"7867da134927ed41114f4b6ff71729ab218e669c5c679a62739c8fdbecd1276b"} Jun 08 12:03:05.033483 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.033123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerDied","Data":"8051e982608d4b2780be05bec5d9395d0f6e812dcda5e00b81600416c97033b7"} Jun 08 12:03:05.310417 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.310382 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:03:05.411345 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.411309 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls\") pod \"b573d032-6595-49e5-87e8-a6b2b2b2829a\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " Jun 08 12:03:05.411345 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.411352 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls\") pod \"b573d032-6595-49e5-87e8-a6b2b2b2829a\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " Jun 08 12:03:05.411683 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.411376 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b573d032-6595-49e5-87e8-a6b2b2b2829a-feast-data\") pod \"b573d032-6595-49e5-87e8-a6b2b2b2829a\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " Jun 08 12:03:05.411683 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.411506 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/b573d032-6595-49e5-87e8-a6b2b2b2829a-kube-api-access-8j66l\") pod \"b573d032-6595-49e5-87e8-a6b2b2b2829a\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " Jun 08 12:03:05.411683 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.411553 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls\") pod \"b573d032-6595-49e5-87e8-a6b2b2b2829a\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " Jun 08 12:03:05.411683 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.411590 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls\") pod \"b573d032-6595-49e5-87e8-a6b2b2b2829a\" (UID: \"b573d032-6595-49e5-87e8-a6b2b2b2829a\") " Jun 08 12:03:05.412000 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.411964 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b573d032-6595-49e5-87e8-a6b2b2b2829a-feast-data" (OuterVolumeSpecName: "feast-data") pod "b573d032-6595-49e5-87e8-a6b2b2b2829a" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:03:05.413914 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.413885 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b573d032-6595-49e5-87e8-a6b2b2b2829a-kube-api-access-8j66l" (OuterVolumeSpecName: "kube-api-access-8j66l") pod "b573d032-6595-49e5-87e8-a6b2b2b2829a" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a"). InnerVolumeSpecName "kube-api-access-8j66l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:03:05.413914 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.413894 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b573d032-6595-49e5-87e8-a6b2b2b2829a" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:03:05.413914 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.413908 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls" (OuterVolumeSpecName: "online-tls") pod "b573d032-6595-49e5-87e8-a6b2b2b2829a" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:03:05.414179 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.414163 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b573d032-6595-49e5-87e8-a6b2b2b2829a" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:03:05.414228 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.414213 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b573d032-6595-49e5-87e8-a6b2b2b2829a" (UID: "b573d032-6595-49e5-87e8-a6b2b2b2829a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:03:05.512544 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.512498 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j66l\" (UniqueName: \"kubernetes.io/projected/b573d032-6595-49e5-87e8-a6b2b2b2829a-kube-api-access-8j66l\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:03:05.512544 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.512534 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:03:05.512776 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.512565 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:03:05.512776 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.512579 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:03:05.512776 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.512592 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b573d032-6595-49e5-87e8-a6b2b2b2829a-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:03:05.512776 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:05.512601 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b573d032-6595-49e5-87e8-a6b2b2b2829a-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:03:06.039745 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.039702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" event={"ID":"b573d032-6595-49e5-87e8-a6b2b2b2829a","Type":"ContainerDied","Data":"dd611067e84a3243a420fe81a12a4d926858811f763cc7ac7ef8a32cd5300ed2"} Jun 08 12:03:06.039745 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.039729 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7" Jun 08 12:03:06.040220 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.039765 2562 scope.go:117] "RemoveContainer" containerID="200dffe55e217af969877d95569c858f987e5bba840fc42329a335a525701084" Jun 08 12:03:06.049671 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.049654 2562 scope.go:117] "RemoveContainer" containerID="7867da134927ed41114f4b6ff71729ab218e669c5c679a62739c8fdbecd1276b" Jun 08 12:03:06.057275 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.057257 2562 scope.go:117] "RemoveContainer" containerID="73152133f20e46a127acf99d5cf5b3bff8e21ee1686633de3d8355b99fa3eaca" Jun 08 12:03:06.062261 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.062239 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7"] Jun 08 12:03:06.065260 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.065235 2562 scope.go:117] "RemoveContainer" containerID="8051e982608d4b2780be05bec5d9395d0f6e812dcda5e00b81600416c97033b7" Jun 08 12:03:06.066299 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.066193 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-95rj7"] Jun 08 12:03:06.072770 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.072758 2562 scope.go:117] "RemoveContainer" containerID="dca8461a82f852fc15fd4f764760b1b0c8f24d520f5c19374bbaf1660cbf256e" Jun 08 12:03:06.079593 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:06.079577 2562 scope.go:117] "RemoveContainer" containerID="f3141a0ccf498fdaae1f71adeeddba03c64c7552d50d7ead23524d73412913b2" Jun 08 12:03:07.202596 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:07.202559 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" path="/var/lib/kubelet/pods/b573d032-6595-49e5-87e8-a6b2b2b2829a/volumes" Jun 08 12:03:56.454989 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:56.454955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:03:56.558283 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:56.558249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:03:56.787841 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:56.787816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:03:57.095395 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095285 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl"] Jun 08 12:03:57.095690 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095672 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095692 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095705 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="registry" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095711 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="registry" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095719 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="offline" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095724 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="offline" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095731 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="feast-init" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095737 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="feast-init" Jun 08 12:03:57.095752 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095751 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="feast-apply" Jun 08 12:03:57.096012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095761 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="feast-apply" Jun 08 12:03:57.096012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095771 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" Jun 08 12:03:57.096012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095776 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" Jun 08 12:03:57.096012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095847 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="registry" Jun 08 12:03:57.096012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095853 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="ui" Jun 08 12:03:57.096012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095862 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="online" Jun 08 12:03:57.096012 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.095867 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b573d032-6595-49e5-87e8-a6b2b2b2829a" containerName="offline" Jun 08 12:03:57.098979 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.098962 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.106403 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.106373 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 08 12:03:57.106535 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.106415 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 08 12:03:57.108145 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.108121 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 08 12:03:57.108256 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.108151 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 08 12:03:57.108256 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.108157 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-7wvj2\"" Jun 08 12:03:57.108256 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.108157 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 08 12:03:57.110649 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.110633 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 08 12:03:57.113712 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.113687 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:03:57.211740 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.211708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.211937 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.211761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzzq\" (UniqueName: \"kubernetes.io/projected/5d85b6b4-999c-49fa-ab2b-aabb76c47130-kube-api-access-wbzzq\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.211937 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.211792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.211937 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.211809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d85b6b4-999c-49fa-ab2b-aabb76c47130-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.211937 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.211825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5d85b6b4-999c-49fa-ab2b-aabb76c47130-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.211937 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.211839 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.253322 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.253282 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl"] Jun 08 12:03:57.313148 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.313106 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzzq\" (UniqueName: \"kubernetes.io/projected/5d85b6b4-999c-49fa-ab2b-aabb76c47130-kube-api-access-wbzzq\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.313358 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.313156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.313358 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.313177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d85b6b4-999c-49fa-ab2b-aabb76c47130-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.313358 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.313301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5d85b6b4-999c-49fa-ab2b-aabb76c47130-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.313358 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.313343 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.313681 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.313421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.313681 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.313528 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d85b6b4-999c-49fa-ab2b-aabb76c47130-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.314025 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.314003 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5d85b6b4-999c-49fa-ab2b-aabb76c47130-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.315848 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.315819 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.315944 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.315900 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.315985 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.315954 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.332912 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.332880 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzzq\" (UniqueName: \"kubernetes.io/projected/5d85b6b4-999c-49fa-ab2b-aabb76c47130-kube-api-access-wbzzq\") pod \"feast-simple-feast-remote-setup-669bcf45fc-55xbl\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.347027 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.346941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:03:57.408270 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.408229 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:03:57.582068 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:03:57.582037 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d85b6b4_999c_49fa_ab2b_aabb76c47130.slice/crio-06732da551ff367d961bbcb96d7fd84d314b74f7c131bb342d1e065c82548eb0 WatchSource:0}: Error finding container 06732da551ff367d961bbcb96d7fd84d314b74f7c131bb342d1e065c82548eb0: Status 404 returned error can't find the container with id 06732da551ff367d961bbcb96d7fd84d314b74f7c131bb342d1e065c82548eb0 Jun 08 12:03:57.582419 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:57.582331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl"] Jun 08 12:03:58.250132 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:58.250094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerStarted","Data":"5af120e78f1209597f57aaaad5c632719ac081690d0f60b66aa69b9b46773511"} Jun 08 12:03:58.250315 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:03:58.250147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerStarted","Data":"06732da551ff367d961bbcb96d7fd84d314b74f7c131bb342d1e065c82548eb0"} Jun 08 12:04:01.259999 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:01.259971 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 12:04:01.271625 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:01.271587 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 12:04:02.268360 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:02.268329 2562 generic.go:356] "Generic (PLEG): container finished" podID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerID="5af120e78f1209597f57aaaad5c632719ac081690d0f60b66aa69b9b46773511" exitCode=0 Jun 08 12:04:02.268773 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:02.268371 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerDied","Data":"5af120e78f1209597f57aaaad5c632719ac081690d0f60b66aa69b9b46773511"} Jun 08 12:04:03.273684 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:03.273650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerStarted","Data":"2fca75a12e52e60285b50b7477357bad779c87e4dcf6798cf0840c7d2e7cbdc6"} Jun 08 12:04:08.299867 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:08.299830 2562 generic.go:356] "Generic (PLEG): container finished" podID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerID="2fca75a12e52e60285b50b7477357bad779c87e4dcf6798cf0840c7d2e7cbdc6" exitCode=0 Jun 08 12:04:08.299867 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:08.299869 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerDied","Data":"2fca75a12e52e60285b50b7477357bad779c87e4dcf6798cf0840c7d2e7cbdc6"} Jun 08 12:04:09.307008 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.306968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerStarted","Data":"ec0bcfa2d365b49186097a506c4a19f0edc5e64f0052b4989c7c4acfa96003bb"} Jun 08 12:04:09.307500 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.307032 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerStarted","Data":"5bcdd8e084c35978466eee12c66d0a3aae6bd9c3120b021da4cac1b3fffa35a1"} Jun 08 12:04:09.307500 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.307051 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerStarted","Data":"c20e523fc4192e3e9102e857c9bfec3302032224d3fc4e8a9768b24bfaf46cdc"} Jun 08 12:04:09.352828 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.352765 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podStartSLOduration=12.352746548 podStartE2EDuration="12.352746548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 12:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:04:09.349659073 +0000 UTC m=+1508.651435771" watchObservedRunningTime="2026-06-08 12:04:09.352746548 +0000 UTC m=+1508.654523245" Jun 08 12:04:09.408646 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.408584 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:09.408931 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.408905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:09.409041 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.408939 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:09.410441 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.410385 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused" Jun 08 12:04:09.410441 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.410420 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 12:04:09.410585 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:09.410438 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 12:04:12.408886 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:12.408844 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 12:04:12.409379 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:12.408843 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 12:04:12.409379 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:12.408838 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused" Jun 08 12:04:15.409690 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:15.409660 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:15.410104 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:15.409776 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:15.410191 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:15.410168 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:15.410253 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:15.410215 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:15.410399 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:15.410369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:15.410584 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:15.410572 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:15.413770 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:15.413750 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:16.340507 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:16.340471 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:04:16.343656 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:04:16.343631 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:05:18.188215 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.188173 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl"] Jun 08 12:05:18.190913 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.188470 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" containerID="cri-o://c20e523fc4192e3e9102e857c9bfec3302032224d3fc4e8a9768b24bfaf46cdc" gracePeriod=30 Jun 08 12:05:18.190913 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.188552 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" containerID="cri-o://ec0bcfa2d365b49186097a506c4a19f0edc5e64f0052b4989c7c4acfa96003bb" gracePeriod=30 Jun 08 12:05:18.190913 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.188583 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="offline" containerID="cri-o://5bcdd8e084c35978466eee12c66d0a3aae6bd9c3120b021da4cac1b3fffa35a1" gracePeriod=30 Jun 08 12:05:18.200955 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.200917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:05:18.363408 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.363374 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28"] Jun 08 12:05:18.363844 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.363645 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="registry" containerID="cri-o://99a4ad616ad47b6bbcedf762447488803742ebaafce6121a876c68a7176004f1" gracePeriod=30 Jun 08 12:05:18.363844 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.363737 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" containerID="cri-o://89d6042c6a2adb89c0da1dd2a8d4d8aa0fcab39bf038ffddb65390bab7ad743c" gracePeriod=30 Jun 08 12:05:18.363844 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.363773 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" containerID="cri-o://231a68779f722406f6e19d8c9b1967888b06303ddfad91b84d41d218d806d34b" gracePeriod=30 Jun 08 12:05:18.364122 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:18.363748 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="offline" containerID="cri-o://19699d9723ebf38700f320abfc2e23bd305790abf3dd23da7a6877cf1f412690" gracePeriod=30 Jun 08 12:05:19.507804 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.507712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:05:19.598264 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.598228 2562 generic.go:356] "Generic (PLEG): container finished" podID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerID="ec0bcfa2d365b49186097a506c4a19f0edc5e64f0052b4989c7c4acfa96003bb" exitCode=0 Jun 08 12:05:19.598264 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.598255 2562 generic.go:356] "Generic (PLEG): container finished" podID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerID="c20e523fc4192e3e9102e857c9bfec3302032224d3fc4e8a9768b24bfaf46cdc" exitCode=0 Jun 08 12:05:19.598536 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.598301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerDied","Data":"ec0bcfa2d365b49186097a506c4a19f0edc5e64f0052b4989c7c4acfa96003bb"} Jun 08 12:05:19.598536 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.598343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerDied","Data":"c20e523fc4192e3e9102e857c9bfec3302032224d3fc4e8a9768b24bfaf46cdc"} Jun 08 12:05:19.601119 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.601095 2562 generic.go:356] "Generic (PLEG): container finished" podID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerID="89d6042c6a2adb89c0da1dd2a8d4d8aa0fcab39bf038ffddb65390bab7ad743c" exitCode=0 Jun 08 12:05:19.601119 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.601116 2562 generic.go:356] "Generic (PLEG): container finished" podID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerID="231a68779f722406f6e19d8c9b1967888b06303ddfad91b84d41d218d806d34b" exitCode=0 Jun 08 12:05:19.601265 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.601148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerDied","Data":"89d6042c6a2adb89c0da1dd2a8d4d8aa0fcab39bf038ffddb65390bab7ad743c"} Jun 08 12:05:19.601265 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:19.601178 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerDied","Data":"231a68779f722406f6e19d8c9b1967888b06303ddfad91b84d41d218d806d34b"} Jun 08 12:05:23.517599 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:23.517562 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:05:23.897277 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:23.897231 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 12:05:23.988512 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:23.988472 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 12:05:25.410695 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:25.410640 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 12:05:26.341267 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:26.341218 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 12:05:33.897664 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:33.897571 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 12:05:33.988426 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:33.988380 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 12:05:35.410952 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:35.410903 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 12:05:36.341379 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:36.341334 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 12:05:43.897452 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:43.897401 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Jun 08 12:05:43.897901 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:43.897580 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:05:43.987880 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:43.987840 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Jun 08 12:05:43.988053 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:43.987948 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:05:45.410603 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:45.410559 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Jun 08 12:05:45.411025 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:45.410724 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:05:46.340814 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:46.340762 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Jun 08 12:05:46.340999 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:46.340871 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:05:48.723455 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.723425 2562 generic.go:356] "Generic (PLEG): container finished" podID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerID="5bcdd8e084c35978466eee12c66d0a3aae6bd9c3120b021da4cac1b3fffa35a1" exitCode=137 Jun 08 12:05:48.723906 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.723493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerDied","Data":"5bcdd8e084c35978466eee12c66d0a3aae6bd9c3120b021da4cac1b3fffa35a1"} Jun 08 12:05:48.726388 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.726359 2562 generic.go:356] "Generic (PLEG): container finished" podID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerID="19699d9723ebf38700f320abfc2e23bd305790abf3dd23da7a6877cf1f412690" exitCode=137 Jun 08 12:05:48.726388 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.726384 2562 generic.go:356] "Generic (PLEG): container finished" podID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerID="99a4ad616ad47b6bbcedf762447488803742ebaafce6121a876c68a7176004f1" exitCode=137 Jun 08 12:05:48.726577 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.726403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerDied","Data":"19699d9723ebf38700f320abfc2e23bd305790abf3dd23da7a6877cf1f412690"} Jun 08 12:05:48.726577 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.726430 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerDied","Data":"99a4ad616ad47b6bbcedf762447488803742ebaafce6121a876c68a7176004f1"} Jun 08 12:05:48.837267 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.837243 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:05:48.946084 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.946050 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-ui-tls\") pod \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " Jun 08 12:05:48.946234 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.946109 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5d85b6b4-999c-49fa-ab2b-aabb76c47130-registry-tls\") pod \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " Jun 08 12:05:48.946234 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.946136 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-online-tls\") pod \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " Jun 08 12:05:48.946234 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.946172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-offline-tls\") pod \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " Jun 08 12:05:48.946431 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.946238 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbzzq\" (UniqueName: \"kubernetes.io/projected/5d85b6b4-999c-49fa-ab2b-aabb76c47130-kube-api-access-wbzzq\") pod \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " Jun 08 12:05:48.946431 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.946301 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d85b6b4-999c-49fa-ab2b-aabb76c47130-feast-data\") pod \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\" (UID: \"5d85b6b4-999c-49fa-ab2b-aabb76c47130\") " Jun 08 12:05:48.946763 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.946738 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d85b6b4-999c-49fa-ab2b-aabb76c47130-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5d85b6b4-999c-49fa-ab2b-aabb76c47130" (UID: "5d85b6b4-999c-49fa-ab2b-aabb76c47130"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 12:05:48.947137 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.947022 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d85b6b4-999c-49fa-ab2b-aabb76c47130-feast-data" (OuterVolumeSpecName: "feast-data") pod "5d85b6b4-999c-49fa-ab2b-aabb76c47130" (UID: "5d85b6b4-999c-49fa-ab2b-aabb76c47130"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:05:48.947600 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.947569 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5d85b6b4-999c-49fa-ab2b-aabb76c47130-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:48.947786 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.947773 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5d85b6b4-999c-49fa-ab2b-aabb76c47130-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:48.949273 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.949237 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5d85b6b4-999c-49fa-ab2b-aabb76c47130" (UID: "5d85b6b4-999c-49fa-ab2b-aabb76c47130"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:05:48.949490 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.949464 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5d85b6b4-999c-49fa-ab2b-aabb76c47130" (UID: "5d85b6b4-999c-49fa-ab2b-aabb76c47130"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:05:48.949982 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.949954 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d85b6b4-999c-49fa-ab2b-aabb76c47130-kube-api-access-wbzzq" (OuterVolumeSpecName: "kube-api-access-wbzzq") pod "5d85b6b4-999c-49fa-ab2b-aabb76c47130" (UID: "5d85b6b4-999c-49fa-ab2b-aabb76c47130"). InnerVolumeSpecName "kube-api-access-wbzzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:05:48.950095 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:48.950046 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-online-tls" (OuterVolumeSpecName: "online-tls") pod "5d85b6b4-999c-49fa-ab2b-aabb76c47130" (UID: "5d85b6b4-999c-49fa-ab2b-aabb76c47130"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:05:49.008589 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.008566 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:05:49.048623 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.048586 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.048823 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.048630 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzzq\" (UniqueName: \"kubernetes.io/projected/5d85b6b4-999c-49fa-ab2b-aabb76c47130-kube-api-access-wbzzq\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.048823 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.048642 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.048823 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.048651 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5d85b6b4-999c-49fa-ab2b-aabb76c47130-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.149289 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.149251 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-offline-tls\") pod \"bed66c42-16bd-4f78-8a3e-a694f44b606e\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " Jun 08 12:05:49.149289 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.149297 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjfxb\" (UniqueName: \"kubernetes.io/projected/bed66c42-16bd-4f78-8a3e-a694f44b606e-kube-api-access-hjfxb\") pod \"bed66c42-16bd-4f78-8a3e-a694f44b606e\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " Jun 08 12:05:49.149508 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.149324 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls\") pod \"bed66c42-16bd-4f78-8a3e-a694f44b606e\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " Jun 08 12:05:49.149508 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.149345 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-registry-tls\") pod \"bed66c42-16bd-4f78-8a3e-a694f44b606e\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " Jun 08 12:05:49.149577 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.149528 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bed66c42-16bd-4f78-8a3e-a694f44b606e-feast-data\") pod \"bed66c42-16bd-4f78-8a3e-a694f44b606e\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " Jun 08 12:05:49.149673 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.149657 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls\") pod \"bed66c42-16bd-4f78-8a3e-a694f44b606e\" (UID: \"bed66c42-16bd-4f78-8a3e-a694f44b606e\") " Jun 08 12:05:49.150075 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.150048 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed66c42-16bd-4f78-8a3e-a694f44b606e-feast-data" (OuterVolumeSpecName: "feast-data") pod "bed66c42-16bd-4f78-8a3e-a694f44b606e" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:05:49.151631 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.151577 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bed66c42-16bd-4f78-8a3e-a694f44b606e" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:05:49.151733 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.151642 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed66c42-16bd-4f78-8a3e-a694f44b606e-kube-api-access-hjfxb" (OuterVolumeSpecName: "kube-api-access-hjfxb") pod "bed66c42-16bd-4f78-8a3e-a694f44b606e" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e"). InnerVolumeSpecName "kube-api-access-hjfxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:05:49.151960 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.151940 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls" (OuterVolumeSpecName: "online-tls") pod "bed66c42-16bd-4f78-8a3e-a694f44b606e" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:05:49.152071 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.152054 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bed66c42-16bd-4f78-8a3e-a694f44b606e" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:05:49.152111 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.152073 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bed66c42-16bd-4f78-8a3e-a694f44b606e" (UID: "bed66c42-16bd-4f78-8a3e-a694f44b606e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:05:49.250938 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.250872 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.250938 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.250896 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.250938 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.250907 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjfxb\" (UniqueName: \"kubernetes.io/projected/bed66c42-16bd-4f78-8a3e-a694f44b606e-kube-api-access-hjfxb\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.250938 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.250915 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.250938 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.250923 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bed66c42-16bd-4f78-8a3e-a694f44b606e-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.250938 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.250932 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bed66c42-16bd-4f78-8a3e-a694f44b606e-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:05:49.732384 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.732347 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" event={"ID":"5d85b6b4-999c-49fa-ab2b-aabb76c47130","Type":"ContainerDied","Data":"06732da551ff367d961bbcb96d7fd84d314b74f7c131bb342d1e065c82548eb0"} Jun 08 12:05:49.732384 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.732378 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl" Jun 08 12:05:49.732961 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.732408 2562 scope.go:117] "RemoveContainer" containerID="ec0bcfa2d365b49186097a506c4a19f0edc5e64f0052b4989c7c4acfa96003bb" Jun 08 12:05:49.735783 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.735715 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" event={"ID":"bed66c42-16bd-4f78-8a3e-a694f44b606e","Type":"ContainerDied","Data":"91200fb89f2875a3012eec54287766c4b7d5c027756b478d9d07c2a7fda466e3"} Jun 08 12:05:49.735783 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.735755 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28" Jun 08 12:05:49.742480 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.742462 2562 scope.go:117] "RemoveContainer" containerID="5bcdd8e084c35978466eee12c66d0a3aae6bd9c3120b021da4cac1b3fffa35a1" Jun 08 12:05:49.750072 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.750054 2562 scope.go:117] "RemoveContainer" containerID="c20e523fc4192e3e9102e857c9bfec3302032224d3fc4e8a9768b24bfaf46cdc" Jun 08 12:05:49.754114 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.754091 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28"] Jun 08 12:05:49.757906 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.757886 2562 scope.go:117] "RemoveContainer" containerID="2fca75a12e52e60285b50b7477357bad779c87e4dcf6798cf0840c7d2e7cbdc6" Jun 08 12:05:49.760460 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.760436 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-zsk28"] Jun 08 12:05:49.765911 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.765891 2562 scope.go:117] "RemoveContainer" containerID="5af120e78f1209597f57aaaad5c632719ac081690d0f60b66aa69b9b46773511" Jun 08 12:05:49.773049 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.773024 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl"] Jun 08 12:05:49.773405 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.773394 2562 scope.go:117] "RemoveContainer" containerID="89d6042c6a2adb89c0da1dd2a8d4d8aa0fcab39bf038ffddb65390bab7ad743c" Jun 08 12:05:49.779401 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.779379 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-55xbl"] Jun 08 12:05:49.783843 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.783825 2562 scope.go:117] "RemoveContainer" containerID="19699d9723ebf38700f320abfc2e23bd305790abf3dd23da7a6877cf1f412690" Jun 08 12:05:49.791330 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.791310 2562 scope.go:117] "RemoveContainer" containerID="231a68779f722406f6e19d8c9b1967888b06303ddfad91b84d41d218d806d34b" Jun 08 12:05:49.801521 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.801492 2562 scope.go:117] "RemoveContainer" containerID="99a4ad616ad47b6bbcedf762447488803742ebaafce6121a876c68a7176004f1" Jun 08 12:05:49.809415 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.809394 2562 scope.go:117] "RemoveContainer" containerID="9c8aed27c7b2541ad447790360c8614277d25f45ef9ebe4cef82b0346722a836" Jun 08 12:05:49.816468 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:49.816451 2562 scope.go:117] "RemoveContainer" containerID="502169d466256fe669d4ca86354e85451c23e3c07582cdba06c204d9d1bc0c9f" Jun 08 12:05:51.202830 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:51.202799 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" path="/var/lib/kubelet/pods/5d85b6b4-999c-49fa-ab2b-aabb76c47130/volumes" Jun 08 12:05:51.203325 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:05:51.203312 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" path="/var/lib/kubelet/pods/bed66c42-16bd-4f78-8a3e-a694f44b606e/volumes" Jun 08 12:06:01.251670 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:01.251602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:01.360154 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:01.360123 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:06.975491 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:06.975449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:07.077376 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:07.077342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:11.935895 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:11.935828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:26.110228 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:26.110172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:26.207969 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:26.207940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:31.014393 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.014351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:31.121574 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.121543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:31.135469 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135435 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg"] Jun 08 12:06:31.135831 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135800 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" Jun 08 12:06:31.135831 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135818 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135839 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="feast-apply" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135847 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="feast-apply" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135858 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135866 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135880 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="feast-apply" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135890 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="feast-apply" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135898 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135907 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135921 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="offline" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135929 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="offline" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135941 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="feast-init" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135950 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="feast-init" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135960 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135969 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135978 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="feast-init" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.135989 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="feast-init" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136008 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="registry" Jun 08 12:06:31.136013 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136016 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="registry" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136025 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="offline" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136033 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="offline" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136107 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="offline" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136121 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="offline" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136132 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="online" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136140 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="online" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136150 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="ui" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136159 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d85b6b4-999c-49fa-ab2b-aabb76c47130" containerName="ui" Jun 08 12:06:31.136964 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.136168 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed66c42-16bd-4f78-8a3e-a694f44b606e" containerName="registry" Jun 08 12:06:31.144476 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.144456 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.146731 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.146696 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 08 12:06:31.146731 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.146716 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 08 12:06:31.146731 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.146727 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 08 12:06:31.147011 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.146995 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 08 12:06:31.147215 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.147197 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-pqxn4\"" Jun 08 12:06:31.147270 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.147208 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 08 12:06:31.147270 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.147224 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 08 12:06:31.153853 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.153823 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg"] Jun 08 12:06:31.183091 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.183061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.183091 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.183091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.183244 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.183125 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.183244 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.183210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fead9023-fb04-4783-9fe0-6ee9669815e5-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.183244 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.183237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.183349 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.183256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8qf\" (UniqueName: \"kubernetes.io/projected/fead9023-fb04-4783-9fe0-6ee9669815e5-kube-api-access-6x8qf\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.215529 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.215504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:31.283730 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.283705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.283842 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.283746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fead9023-fb04-4783-9fe0-6ee9669815e5-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.283842 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.283764 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.283842 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.283790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8qf\" (UniqueName: \"kubernetes.io/projected/fead9023-fb04-4783-9fe0-6ee9669815e5-kube-api-access-6x8qf\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.284001 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.283838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.284001 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.283865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.284001 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:06:31.283868 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jun 08 12:06:31.284001 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:06:31.283940 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls podName:fead9023-fb04-4783-9fe0-6ee9669815e5 nodeName:}" failed. No retries permitted until 2026-06-08 12:06:31.783918929 +0000 UTC m=+1651.085695606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls") pod "feast-simple-feast-setup-75c64c888b-4kkjg" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5") : secret "feast-simple-feast-setup-online-tls" not found Jun 08 12:06:31.284001 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:06:31.283973 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:06:31.284001 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:06:31.284000 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 08 12:06:31.284251 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:06:31.284044 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls podName:fead9023-fb04-4783-9fe0-6ee9669815e5 nodeName:}" failed. No retries permitted until 2026-06-08 12:06:31.784025878 +0000 UTC m=+1651.085802573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls") pod "feast-simple-feast-setup-75c64c888b-4kkjg" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:06:31.284251 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:06:31.284064 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls podName:fead9023-fb04-4783-9fe0-6ee9669815e5 nodeName:}" failed. No retries permitted until 2026-06-08 12:06:31.784055005 +0000 UTC m=+1651.085831696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls") pod "feast-simple-feast-setup-75c64c888b-4kkjg" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5") : secret "feast-simple-feast-setup-offline-tls" not found Jun 08 12:06:31.284251 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.284213 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fead9023-fb04-4783-9fe0-6ee9669815e5-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.286203 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.286184 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.292707 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.292684 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8qf\" (UniqueName: \"kubernetes.io/projected/fead9023-fb04-4783-9fe0-6ee9669815e5-kube-api-access-6x8qf\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.356359 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.356329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:31.455252 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.455224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:31.649508 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.649422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:06:31.787630 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.787578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.787792 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.787649 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.787792 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.787685 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.790159 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.790127 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.790290 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.790166 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:31.790290 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:31.790269 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-4kkjg\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:32.067843 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:32.067808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:32.208390 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:32.208353 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg"] Jun 08 12:06:32.211221 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:06:32.211191 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfead9023_fb04_4783_9fe0_6ee9669815e5.slice/crio-0bd358527980b957ad724c4f5e52e10b7442a8f6a096674333d2d992062a8480 WatchSource:0}: Error finding container 0bd358527980b957ad724c4f5e52e10b7442a8f6a096674333d2d992062a8480: Status 404 returned error can't find the container with id 0bd358527980b957ad724c4f5e52e10b7442a8f6a096674333d2d992062a8480 Jun 08 12:06:32.896094 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:32.896058 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerStarted","Data":"6c2a10065b45bf86164089b9b2400245e79ba3165db631a3195021dd6569cfac"} Jun 08 12:06:32.896284 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:32.896103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerStarted","Data":"0bd358527980b957ad724c4f5e52e10b7442a8f6a096674333d2d992062a8480"} Jun 08 12:06:36.913065 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:36.913032 2562 generic.go:356] "Generic (PLEG): container finished" podID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerID="6c2a10065b45bf86164089b9b2400245e79ba3165db631a3195021dd6569cfac" exitCode=0 Jun 08 12:06:36.913459 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:36.913100 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerDied","Data":"6c2a10065b45bf86164089b9b2400245e79ba3165db631a3195021dd6569cfac"} Jun 08 12:06:37.919294 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:37.919260 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerStarted","Data":"cad0e55a456121bf60e369f3609c0c2e40dad3e2cb3e1589ef1c2f2cc8e9b528"} Jun 08 12:06:42.939841 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:42.939801 2562 generic.go:356] "Generic (PLEG): container finished" podID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerID="cad0e55a456121bf60e369f3609c0c2e40dad3e2cb3e1589ef1c2f2cc8e9b528" exitCode=0 Jun 08 12:06:42.940257 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:42.939862 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerDied","Data":"cad0e55a456121bf60e369f3609c0c2e40dad3e2cb3e1589ef1c2f2cc8e9b528"} Jun 08 12:06:43.949185 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:43.949139 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerStarted","Data":"031583cbd6817807998ce68c11203aa25db41543d62ddf62f55534f83c4b1d74"} Jun 08 12:06:43.949678 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:43.949210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerStarted","Data":"9164cd1a78f1fc08eafb69c23929b08902889afeb7a84801df723ae81ed8068a"} Jun 08 12:06:43.949678 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:43.949235 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerStarted","Data":"597a4f7f153eaaf85aa5eb097c034b753c880d0c29fb2588a0890fb72f8f4a03"} Jun 08 12:06:43.949678 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:43.949255 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerStarted","Data":"0948c1228ec970b255327074aa221c05520a88c1d1bcb0af9ee25d1b4ce282e7"} Jun 08 12:06:43.973891 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:43.973734 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podStartSLOduration=12.973706949 podStartE2EDuration="12.973706949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 12:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:06:43.972091012 +0000 UTC m=+1663.273867736" watchObservedRunningTime="2026-06-08 12:06:43.973706949 +0000 UTC m=+1663.275483648" Jun 08 12:06:44.068289 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.068244 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:44.068488 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.068310 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:44.068488 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.068331 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:44.068488 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.068348 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:44.070514 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.070473 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Jun 08 12:06:44.070693 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.070508 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 12:06:44.070693 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.070521 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 12:06:44.070693 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:44.070587 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Jun 08 12:06:47.069215 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:47.069161 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Jun 08 12:06:47.069842 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:47.069161 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Jun 08 12:06:47.070052 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:47.069170 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 12:06:47.070197 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:47.069366 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 12:06:50.069471 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.069435 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.070118 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.069993 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.070118 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.070109 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.070236 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.070124 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.070236 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.070170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.074140 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.074115 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.981817 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.981777 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.982049 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.981837 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.982049 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.981857 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.982252 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.982232 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.982396 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.982369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:06:50.984770 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:06:50.984754 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:07:52.114525 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.114488 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg"] Jun 08 12:07:52.115042 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.114800 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="registry" containerID="cri-o://0948c1228ec970b255327074aa221c05520a88c1d1bcb0af9ee25d1b4ce282e7" gracePeriod=30 Jun 08 12:07:52.115042 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.114842 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="offline" containerID="cri-o://9164cd1a78f1fc08eafb69c23929b08902889afeb7a84801df723ae81ed8068a" gracePeriod=30 Jun 08 12:07:52.115042 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.114829 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" containerID="cri-o://031583cbd6817807998ce68c11203aa25db41543d62ddf62f55534f83c4b1d74" gracePeriod=30 Jun 08 12:07:52.115042 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.114976 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" containerID="cri-o://597a4f7f153eaaf85aa5eb097c034b753c880d0c29fb2588a0890fb72f8f4a03" gracePeriod=30 Jun 08 12:07:52.117464 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.117440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:52.210423 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.210383 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:52.312734 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.312682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:52.391647 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.391530 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5"] Jun 08 12:07:52.392415 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.392386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:52.395382 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.395365 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.397735 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.397717 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-mgrmz\"" Jun 08 12:07:52.409804 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.409775 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5"] Jun 08 12:07:52.435981 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.435950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e564e669-bf99-42d6-93fa-cec8f1d2406f-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.435981 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.435983 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwgh\" (UniqueName: \"kubernetes.io/projected/e564e669-bf99-42d6-93fa-cec8f1d2406f-kube-api-access-4jwgh\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.436176 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.436007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.436176 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.436115 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.436176 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.436143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.436176 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.436175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.536881 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.536846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.536881 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.536889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.537156 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:07:52.537010 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:07:52.537156 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:07:52.537021 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 08 12:07:52.537156 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:07:52.537071 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls podName:e564e669-bf99-42d6-93fa-cec8f1d2406f nodeName:}" failed. No retries permitted until 2026-06-08 12:07:53.037050736 +0000 UTC m=+1732.338827417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls") pod "feast-simple-feast-setup-75c64c888b-jwvl5" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f") : secret "feast-simple-feast-setup-registry-tls" not found Jun 08 12:07:52.537156 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:07:52.537123 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls podName:e564e669-bf99-42d6-93fa-cec8f1d2406f nodeName:}" failed. No retries permitted until 2026-06-08 12:07:53.037105148 +0000 UTC m=+1732.338881839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls") pod "feast-simple-feast-setup-75c64c888b-jwvl5" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f") : secret "feast-simple-feast-setup-offline-tls" not found Jun 08 12:07:52.537156 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.537022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.537424 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.537186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e564e669-bf99-42d6-93fa-cec8f1d2406f-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.537424 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.537218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwgh\" (UniqueName: \"kubernetes.io/projected/e564e669-bf99-42d6-93fa-cec8f1d2406f-kube-api-access-4jwgh\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.537424 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.537255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.537694 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.537664 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e564e669-bf99-42d6-93fa-cec8f1d2406f-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.538906 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.538878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:52.539959 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.539940 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.540299 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.540277 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.548153 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.548129 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwgh\" (UniqueName: \"kubernetes.io/projected/e564e669-bf99-42d6-93fa-cec8f1d2406f-kube-api-access-4jwgh\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:52.634520 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:52.634485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:53.020631 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.020583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:53.041851 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.041825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:53.041955 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.041863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:53.044364 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.044335 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:53.044471 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.044375 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-jwvl5\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:53.122271 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.122196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:07:53.217260 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.217230 2562 generic.go:356] "Generic (PLEG): container finished" podID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerID="031583cbd6817807998ce68c11203aa25db41543d62ddf62f55534f83c4b1d74" exitCode=0 Jun 08 12:07:53.217260 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.217256 2562 generic.go:356] "Generic (PLEG): container finished" podID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerID="597a4f7f153eaaf85aa5eb097c034b753c880d0c29fb2588a0890fb72f8f4a03" exitCode=0 Jun 08 12:07:53.217447 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.217300 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerDied","Data":"031583cbd6817807998ce68c11203aa25db41543d62ddf62f55534f83c4b1d74"} Jun 08 12:07:53.217447 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.217338 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerDied","Data":"597a4f7f153eaaf85aa5eb097c034b753c880d0c29fb2588a0890fb72f8f4a03"} Jun 08 12:07:53.304474 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.304430 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:07:53.449104 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:53.449079 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5"] Jun 08 12:07:53.451001 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:07:53.450971 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode564e669_bf99_42d6_93fa_cec8f1d2406f.slice/crio-42aecffbe7746a02ebc1469b37e25d4c4fcd95301e98b642a2b525bd5b44c143 WatchSource:0}: Error finding container 42aecffbe7746a02ebc1469b37e25d4c4fcd95301e98b642a2b525bd5b44c143: Status 404 returned error can't find the container with id 42aecffbe7746a02ebc1469b37e25d4c4fcd95301e98b642a2b525bd5b44c143 Jun 08 12:07:54.222811 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:54.222774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerStarted","Data":"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d"} Jun 08 12:07:54.223199 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:54.222820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerStarted","Data":"42aecffbe7746a02ebc1469b37e25d4c4fcd95301e98b642a2b525bd5b44c143"} Jun 08 12:07:58.239928 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:58.239891 2562 generic.go:356] "Generic (PLEG): container finished" podID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerID="2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d" exitCode=0 Jun 08 12:07:58.240334 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:58.239964 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerDied","Data":"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d"} Jun 08 12:07:59.245796 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:07:59.245761 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerStarted","Data":"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6"} Jun 08 12:08:00.982703 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:00.982654 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 12:08:00.982703 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:00.982669 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 12:08:04.268902 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:04.268814 2562 generic.go:356] "Generic (PLEG): container finished" podID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerID="493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6" exitCode=0 Jun 08 12:08:04.268902 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:04.268890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerDied","Data":"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6"} Jun 08 12:08:05.279250 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.279200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerStarted","Data":"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff"} Jun 08 12:08:05.279768 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.279270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerStarted","Data":"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6"} Jun 08 12:08:05.279768 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.279293 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerStarted","Data":"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b"} Jun 08 12:08:05.279768 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.279313 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerStarted","Data":"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2"} Jun 08 12:08:05.304721 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.304639 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podStartSLOduration=13.304585566 podStartE2EDuration="13.304585566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 12:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:08:05.303016906 +0000 UTC m=+1744.604793604" watchObservedRunningTime="2026-06-08 12:08:05.304585566 +0000 UTC m=+1744.606362265" Jun 08 12:08:05.305342 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.305088 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:05.305342 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.305245 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:05.305342 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.305277 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:05.305342 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.305313 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:05.307557 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.307513 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Jun 08 12:08:05.307713 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.307573 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Jun 08 12:08:05.307713 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.307640 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 12:08:05.307713 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:05.307579 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 12:08:08.305749 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:08.305700 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Jun 08 12:08:08.306341 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:08.305701 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 12:08:08.306575 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:08.305781 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Jun 08 12:08:08.306575 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:08.305782 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 12:08:10.982240 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:10.982189 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 12:08:10.982240 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:10.982215 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 12:08:11.306252 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.306223 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.306434 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.306338 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.306599 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.306583 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.307102 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.306994 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.307102 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.307036 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.307102 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.307054 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.307305 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.307262 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.307512 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.307492 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.307563 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.307541 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.310955 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.310934 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.312839 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.312820 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:11.315393 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:11.315375 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:08:20.982106 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:20.982059 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Jun 08 12:08:20.982573 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:20.982063 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Jun 08 12:08:20.982573 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:20.982175 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:08:20.982573 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:20.982359 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:08:22.358532 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.358498 2562 generic.go:356] "Generic (PLEG): container finished" podID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerID="9164cd1a78f1fc08eafb69c23929b08902889afeb7a84801df723ae81ed8068a" exitCode=137 Jun 08 12:08:22.358532 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.358522 2562 generic.go:356] "Generic (PLEG): container finished" podID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerID="0948c1228ec970b255327074aa221c05520a88c1d1bcb0af9ee25d1b4ce282e7" exitCode=137 Jun 08 12:08:22.358955 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.358564 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerDied","Data":"9164cd1a78f1fc08eafb69c23929b08902889afeb7a84801df723ae81ed8068a"} Jun 08 12:08:22.358955 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.358602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerDied","Data":"0948c1228ec970b255327074aa221c05520a88c1d1bcb0af9ee25d1b4ce282e7"} Jun 08 12:08:22.769140 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.769114 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:08:22.912104 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.912001 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-ui-tls\") pod \"fead9023-fb04-4783-9fe0-6ee9669815e5\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " Jun 08 12:08:22.912104 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.912049 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x8qf\" (UniqueName: \"kubernetes.io/projected/fead9023-fb04-4783-9fe0-6ee9669815e5-kube-api-access-6x8qf\") pod \"fead9023-fb04-4783-9fe0-6ee9669815e5\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " Jun 08 12:08:22.912104 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.912075 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fead9023-fb04-4783-9fe0-6ee9669815e5-feast-data\") pod \"fead9023-fb04-4783-9fe0-6ee9669815e5\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " Jun 08 12:08:22.912405 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.912189 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls\") pod \"fead9023-fb04-4783-9fe0-6ee9669815e5\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " Jun 08 12:08:22.912405 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.912223 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls\") pod \"fead9023-fb04-4783-9fe0-6ee9669815e5\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " Jun 08 12:08:22.912405 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.912264 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls\") pod \"fead9023-fb04-4783-9fe0-6ee9669815e5\" (UID: \"fead9023-fb04-4783-9fe0-6ee9669815e5\") " Jun 08 12:08:22.912675 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.912648 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fead9023-fb04-4783-9fe0-6ee9669815e5-feast-data" (OuterVolumeSpecName: "feast-data") pod "fead9023-fb04-4783-9fe0-6ee9669815e5" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:08:22.914578 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.914542 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fead9023-fb04-4783-9fe0-6ee9669815e5" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:08:22.914778 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.914755 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fead9023-fb04-4783-9fe0-6ee9669815e5" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:08:22.914845 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.914819 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fead9023-fb04-4783-9fe0-6ee9669815e5-kube-api-access-6x8qf" (OuterVolumeSpecName: "kube-api-access-6x8qf") pod "fead9023-fb04-4783-9fe0-6ee9669815e5" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5"). InnerVolumeSpecName "kube-api-access-6x8qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:08:22.914914 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.914898 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "fead9023-fb04-4783-9fe0-6ee9669815e5" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:08:22.914949 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:22.914916 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls" (OuterVolumeSpecName: "online-tls") pod "fead9023-fb04-4783-9fe0-6ee9669815e5" (UID: "fead9023-fb04-4783-9fe0-6ee9669815e5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:08:23.013569 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.013527 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:08:23.013569 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.013564 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:08:23.013569 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.013572 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:08:23.013569 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.013580 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fead9023-fb04-4783-9fe0-6ee9669815e5-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:08:23.013857 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.013588 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8qf\" (UniqueName: \"kubernetes.io/projected/fead9023-fb04-4783-9fe0-6ee9669815e5-kube-api-access-6x8qf\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:08:23.013857 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.013597 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fead9023-fb04-4783-9fe0-6ee9669815e5-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:08:23.366868 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.366840 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" Jun 08 12:08:23.367297 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.366836 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg" event={"ID":"fead9023-fb04-4783-9fe0-6ee9669815e5","Type":"ContainerDied","Data":"0bd358527980b957ad724c4f5e52e10b7442a8f6a096674333d2d992062a8480"} Jun 08 12:08:23.367297 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.366965 2562 scope.go:117] "RemoveContainer" containerID="031583cbd6817807998ce68c11203aa25db41543d62ddf62f55534f83c4b1d74" Jun 08 12:08:23.377971 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.377939 2562 scope.go:117] "RemoveContainer" containerID="9164cd1a78f1fc08eafb69c23929b08902889afeb7a84801df723ae81ed8068a" Jun 08 12:08:23.386769 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.386742 2562 scope.go:117] "RemoveContainer" containerID="597a4f7f153eaaf85aa5eb097c034b753c880d0c29fb2588a0890fb72f8f4a03" Jun 08 12:08:23.389806 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.389766 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg"] Jun 08 12:08:23.393597 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.393542 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-4kkjg"] Jun 08 12:08:23.397862 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.397840 2562 scope.go:117] "RemoveContainer" containerID="0948c1228ec970b255327074aa221c05520a88c1d1bcb0af9ee25d1b4ce282e7" Jun 08 12:08:23.405805 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.405787 2562 scope.go:117] "RemoveContainer" containerID="cad0e55a456121bf60e369f3609c0c2e40dad3e2cb3e1589ef1c2f2cc8e9b528" Jun 08 12:08:23.413277 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:23.413256 2562 scope.go:117] "RemoveContainer" containerID="6c2a10065b45bf86164089b9b2400245e79ba3165db631a3195021dd6569cfac" Jun 08 12:08:25.202962 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:08:25.202928 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" path="/var/lib/kubelet/pods/fead9023-fb04-4783-9fe0-6ee9669815e5/volumes" Jun 08 12:09:01.284922 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:01.284893 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 12:09:01.296719 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:01.296695 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 12:09:13.441091 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.441052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:09:13.545479 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.545444 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:09:13.700938 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.700841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:09:13.771493 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.771462 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:09:13.775220 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775196 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt"] Jun 08 12:09:13.775521 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775510 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="feast-apply" Jun 08 12:09:13.775568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775523 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="feast-apply" Jun 08 12:09:13.775568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775532 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" Jun 08 12:09:13.775568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775538 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" Jun 08 12:09:13.775568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775549 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="offline" Jun 08 12:09:13.775568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775554 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="offline" Jun 08 12:09:13.775568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775562 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="registry" Jun 08 12:09:13.775568 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775567 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="registry" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775574 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775579 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775589 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="feast-init" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775594 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="feast-init" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775671 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="offline" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775678 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="ui" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775684 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="online" Jun 08 12:09:13.775798 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.775691 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fead9023-fb04-4783-9fe0-6ee9669815e5" containerName="registry" Jun 08 12:09:13.780274 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.780257 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:13.782888 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.782862 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 08 12:09:13.783004 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.782902 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 08 12:09:13.783004 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.782925 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 08 12:09:13.783004 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.782867 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 08 12:09:13.783004 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.782953 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-mv59v\"" Jun 08 12:09:13.783004 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.782983 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 08 12:09:13.783273 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.783093 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 08 12:09:13.791423 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.791400 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt"] Jun 08 12:09:13.855912 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.855878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:09:13.911740 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.911705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:13.911870 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.911749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:13.911870 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.911826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11fc7f98-97ae-4865-9613-b27126b480d4-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:13.911950 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.911872 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc7f98-97ae-4865-9613-b27126b480d4-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:13.911950 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.911893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:13.911950 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.911909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmvl9\" (UniqueName: \"kubernetes.io/projected/11fc7f98-97ae-4865-9613-b27126b480d4-kube-api-access-cmvl9\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:13.958027 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:13.957947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:09:14.013144 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc7f98-97ae-4865-9613-b27126b480d4-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.013144 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013147 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.013371 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmvl9\" (UniqueName: \"kubernetes.io/projected/11fc7f98-97ae-4865-9613-b27126b480d4-kube-api-access-cmvl9\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.013371 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.013371 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.013371 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11fc7f98-97ae-4865-9613-b27126b480d4-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.013596 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013553 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc7f98-97ae-4865-9613-b27126b480d4-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.013998 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.013975 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11fc7f98-97ae-4865-9613-b27126b480d4-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.015900 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.015881 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.016039 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.016020 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.016080 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.016041 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.021447 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.021419 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmvl9\" (UniqueName: \"kubernetes.io/projected/11fc7f98-97ae-4865-9613-b27126b480d4-kube-api-access-cmvl9\") pod \"feast-simple-feast-remote-setup-669bcf45fc-bxxdt\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.058362 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.058331 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:09:14.089828 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.089796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:14.227097 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.227068 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt"] Jun 08 12:09:14.229627 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:09:14.229577 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11fc7f98_97ae_4865_9613_b27126b480d4.slice/crio-aa6fa9c8f000b64ced890a966019fefc5144187c4adf06cb0ec1ae5a150f14cc WatchSource:0}: Error finding container aa6fa9c8f000b64ced890a966019fefc5144187c4adf06cb0ec1ae5a150f14cc: Status 404 returned error can't find the container with id aa6fa9c8f000b64ced890a966019fefc5144187c4adf06cb0ec1ae5a150f14cc Jun 08 12:09:14.567653 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.567589 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerStarted","Data":"213644ef740f5ae0082f9f1a1586bdc4fe363d6a2cbfc6ee3c6d6bde028d6404"} Jun 08 12:09:14.568056 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:14.567662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerStarted","Data":"aa6fa9c8f000b64ced890a966019fefc5144187c4adf06cb0ec1ae5a150f14cc"} Jun 08 12:09:19.591817 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:19.591784 2562 generic.go:356] "Generic (PLEG): container finished" podID="11fc7f98-97ae-4865-9613-b27126b480d4" containerID="213644ef740f5ae0082f9f1a1586bdc4fe363d6a2cbfc6ee3c6d6bde028d6404" exitCode=0 Jun 08 12:09:19.592197 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:19.591840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerDied","Data":"213644ef740f5ae0082f9f1a1586bdc4fe363d6a2cbfc6ee3c6d6bde028d6404"} Jun 08 12:09:20.597937 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:20.597902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerStarted","Data":"21eca81b9fe296343617b3b93db36c30474b7d5a08f28e86ced52aa7f8abeb83"} Jun 08 12:09:25.620860 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:25.620825 2562 generic.go:356] "Generic (PLEG): container finished" podID="11fc7f98-97ae-4865-9613-b27126b480d4" containerID="21eca81b9fe296343617b3b93db36c30474b7d5a08f28e86ced52aa7f8abeb83" exitCode=0 Jun 08 12:09:25.621258 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:25.620899 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerDied","Data":"21eca81b9fe296343617b3b93db36c30474b7d5a08f28e86ced52aa7f8abeb83"} Jun 08 12:09:26.628934 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:26.628891 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerStarted","Data":"3271fa88b0b1e2bbcd557ab5215c9565d8ef1916547d72bba7545780c8f0abe5"} Jun 08 12:09:26.629308 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:26.628961 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerStarted","Data":"69ae7205fe3338de545bdb5b81063bf937dc0bf5612078330f549178565f708e"} Jun 08 12:09:26.629308 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:26.628982 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerStarted","Data":"07ad11610c6ed406ffdff9b060f1cf3558344a7abee469d54b8ccb2d1605fda3"} Jun 08 12:09:26.652206 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:26.652153 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podStartSLOduration=13.652135509 podStartE2EDuration="13.652135509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 12:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:09:26.649783451 +0000 UTC m=+1825.951560150" watchObservedRunningTime="2026-06-08 12:09:26.652135509 +0000 UTC m=+1825.953912205" Jun 08 12:09:29.090384 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:29.090327 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:29.091029 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:29.090997 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:29.091029 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:29.091034 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:29.092179 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:29.092139 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.38:8816: connect: connection refused" Jun 08 12:09:29.092282 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:29.092166 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jun 08 12:09:29.092282 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:29.092195 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jun 08 12:09:32.091390 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.091359 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.091799 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.091476 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.091913 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.091886 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.092039 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.091933 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.092123 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.092107 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.092165 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.092151 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.095585 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.095567 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.659273 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.659242 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:09:32.662267 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:09:32.662242 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:10:34.879160 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.879123 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt"] Jun 08 12:10:34.879708 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.879349 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" containerID="cri-o://07ad11610c6ed406ffdff9b060f1cf3558344a7abee469d54b8ccb2d1605fda3" gracePeriod=30 Jun 08 12:10:34.879708 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.879393 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" containerID="cri-o://3271fa88b0b1e2bbcd557ab5215c9565d8ef1916547d72bba7545780c8f0abe5" gracePeriod=30 Jun 08 12:10:34.879708 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.879394 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="offline" containerID="cri-o://69ae7205fe3338de545bdb5b81063bf937dc0bf5612078330f549178565f708e" gracePeriod=30 Jun 08 12:10:34.886718 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.886682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:10:34.983808 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.983771 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5"] Jun 08 12:10:34.984124 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.984071 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="registry" containerID="cri-o://7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2" gracePeriod=30 Jun 08 12:10:34.984256 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.984113 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" containerID="cri-o://ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b" gracePeriod=30 Jun 08 12:10:34.984256 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.984143 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" containerID="cri-o://35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff" gracePeriod=30 Jun 08 12:10:34.984256 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:34.984113 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="offline" containerID="cri-o://0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6" gracePeriod=30 Jun 08 12:10:35.906551 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.906460 2562 generic.go:356] "Generic (PLEG): container finished" podID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerID="35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff" exitCode=0 Jun 08 12:10:35.906551 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.906486 2562 generic.go:356] "Generic (PLEG): container finished" podID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerID="ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b" exitCode=0 Jun 08 12:10:35.906551 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.906531 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerDied","Data":"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff"} Jun 08 12:10:35.907093 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.906590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerDied","Data":"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b"} Jun 08 12:10:35.909188 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.909165 2562 generic.go:356] "Generic (PLEG): container finished" podID="11fc7f98-97ae-4865-9613-b27126b480d4" containerID="3271fa88b0b1e2bbcd557ab5215c9565d8ef1916547d72bba7545780c8f0abe5" exitCode=0 Jun 08 12:10:35.909188 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.909187 2562 generic.go:356] "Generic (PLEG): container finished" podID="11fc7f98-97ae-4865-9613-b27126b480d4" containerID="07ad11610c6ed406ffdff9b060f1cf3558344a7abee469d54b8ccb2d1605fda3" exitCode=0 Jun 08 12:10:35.909340 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.909206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerDied","Data":"3271fa88b0b1e2bbcd557ab5215c9565d8ef1916547d72bba7545780c8f0abe5"} Jun 08 12:10:35.909340 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:35.909236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerDied","Data":"07ad11610c6ed406ffdff9b060f1cf3558344a7abee469d54b8ccb2d1605fda3"} Jun 08 12:10:36.161397 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:36.161297 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:10:40.144556 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:40.144517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:10:41.307239 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:41.307186 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 12:10:41.313334 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:41.313299 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 12:10:42.092533 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:42.092488 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jun 08 12:10:42.659400 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:42.659352 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jun 08 12:10:51.307412 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:51.307358 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 12:10:51.312847 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:51.312815 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 12:10:52.092439 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:52.092394 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jun 08 12:10:52.659402 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:10:52.659361 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jun 08 12:11:01.307440 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:01.307391 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Jun 08 12:11:01.307940 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:01.307539 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:11:01.312870 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:01.312844 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Jun 08 12:11:01.312959 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:01.312944 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:11:02.092784 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:02.092737 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Jun 08 12:11:02.093061 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:02.092869 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:11:02.659843 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:02.659802 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Jun 08 12:11:02.660229 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:02.659921 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:11:05.022435 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.022403 2562 generic.go:356] "Generic (PLEG): container finished" podID="11fc7f98-97ae-4865-9613-b27126b480d4" containerID="69ae7205fe3338de545bdb5b81063bf937dc0bf5612078330f549178565f708e" exitCode=137 Jun 08 12:11:05.022816 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.022459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerDied","Data":"69ae7205fe3338de545bdb5b81063bf937dc0bf5612078330f549178565f708e"} Jun 08 12:11:05.531508 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.531465 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:11:05.633200 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.633175 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:11:05.647078 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647003 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmvl9\" (UniqueName: \"kubernetes.io/projected/11fc7f98-97ae-4865-9613-b27126b480d4-kube-api-access-cmvl9\") pod \"11fc7f98-97ae-4865-9613-b27126b480d4\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " Jun 08 12:11:05.647078 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647057 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-online-tls\") pod \"11fc7f98-97ae-4865-9613-b27126b480d4\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " Jun 08 12:11:05.647267 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647089 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11fc7f98-97ae-4865-9613-b27126b480d4-registry-tls\") pod \"11fc7f98-97ae-4865-9613-b27126b480d4\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " Jun 08 12:11:05.647267 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647116 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-offline-tls\") pod \"11fc7f98-97ae-4865-9613-b27126b480d4\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " Jun 08 12:11:05.647267 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647136 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-ui-tls\") pod \"11fc7f98-97ae-4865-9613-b27126b480d4\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " Jun 08 12:11:05.647267 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647229 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc7f98-97ae-4865-9613-b27126b480d4-feast-data\") pod \"11fc7f98-97ae-4865-9613-b27126b480d4\" (UID: \"11fc7f98-97ae-4865-9613-b27126b480d4\") " Jun 08 12:11:05.647524 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647497 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fc7f98-97ae-4865-9613-b27126b480d4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "11fc7f98-97ae-4865-9613-b27126b480d4" (UID: "11fc7f98-97ae-4865-9613-b27126b480d4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 08 12:11:05.647999 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.647968 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc7f98-97ae-4865-9613-b27126b480d4-feast-data" (OuterVolumeSpecName: "feast-data") pod "11fc7f98-97ae-4865-9613-b27126b480d4" (UID: "11fc7f98-97ae-4865-9613-b27126b480d4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:11:05.649474 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.649448 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-online-tls" (OuterVolumeSpecName: "online-tls") pod "11fc7f98-97ae-4865-9613-b27126b480d4" (UID: "11fc7f98-97ae-4865-9613-b27126b480d4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:11:05.649555 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.649466 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "11fc7f98-97ae-4865-9613-b27126b480d4" (UID: "11fc7f98-97ae-4865-9613-b27126b480d4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:11:05.649818 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.649798 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "11fc7f98-97ae-4865-9613-b27126b480d4" (UID: "11fc7f98-97ae-4865-9613-b27126b480d4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:11:05.649818 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.649805 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fc7f98-97ae-4865-9613-b27126b480d4-kube-api-access-cmvl9" (OuterVolumeSpecName: "kube-api-access-cmvl9") pod "11fc7f98-97ae-4865-9613-b27126b480d4" (UID: "11fc7f98-97ae-4865-9613-b27126b480d4"). InnerVolumeSpecName "kube-api-access-cmvl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:11:05.748028 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.747990 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls\") pod \"e564e669-bf99-42d6-93fa-cec8f1d2406f\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " Jun 08 12:11:05.748028 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748032 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-online-tls\") pod \"e564e669-bf99-42d6-93fa-cec8f1d2406f\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " Jun 08 12:11:05.748302 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748078 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-ui-tls\") pod \"e564e669-bf99-42d6-93fa-cec8f1d2406f\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " Jun 08 12:11:05.748302 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748124 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e564e669-bf99-42d6-93fa-cec8f1d2406f-feast-data\") pod \"e564e669-bf99-42d6-93fa-cec8f1d2406f\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " Jun 08 12:11:05.748302 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748154 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls\") pod \"e564e669-bf99-42d6-93fa-cec8f1d2406f\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " Jun 08 12:11:05.748302 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748205 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jwgh\" (UniqueName: \"kubernetes.io/projected/e564e669-bf99-42d6-93fa-cec8f1d2406f-kube-api-access-4jwgh\") pod \"e564e669-bf99-42d6-93fa-cec8f1d2406f\" (UID: \"e564e669-bf99-42d6-93fa-cec8f1d2406f\") " Jun 08 12:11:05.748502 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748409 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.748502 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748427 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.748502 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748439 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc7f98-97ae-4865-9613-b27126b480d4-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.748502 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748454 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmvl9\" (UniqueName: \"kubernetes.io/projected/11fc7f98-97ae-4865-9613-b27126b480d4-kube-api-access-cmvl9\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.748502 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748494 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/11fc7f98-97ae-4865-9613-b27126b480d4-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.748782 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748508 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/11fc7f98-97ae-4865-9613-b27126b480d4-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.748782 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.748724 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e564e669-bf99-42d6-93fa-cec8f1d2406f-feast-data" (OuterVolumeSpecName: "feast-data") pod "e564e669-bf99-42d6-93fa-cec8f1d2406f" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:11:05.750432 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.750395 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-online-tls" (OuterVolumeSpecName: "online-tls") pod "e564e669-bf99-42d6-93fa-cec8f1d2406f" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:11:05.750557 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.750510 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e564e669-bf99-42d6-93fa-cec8f1d2406f-kube-api-access-4jwgh" (OuterVolumeSpecName: "kube-api-access-4jwgh") pod "e564e669-bf99-42d6-93fa-cec8f1d2406f" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f"). InnerVolumeSpecName "kube-api-access-4jwgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:11:05.750557 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.750501 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e564e669-bf99-42d6-93fa-cec8f1d2406f" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:11:05.750823 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.750802 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e564e669-bf99-42d6-93fa-cec8f1d2406f" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:11:05.750823 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.750814 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e564e669-bf99-42d6-93fa-cec8f1d2406f" (UID: "e564e669-bf99-42d6-93fa-cec8f1d2406f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 08 12:11:05.849416 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.849362 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-offline-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.849416 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.849410 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwgh\" (UniqueName: \"kubernetes.io/projected/e564e669-bf99-42d6-93fa-cec8f1d2406f-kube-api-access-4jwgh\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.849416 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.849420 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-registry-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.849416 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.849430 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-online-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.849710 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.849437 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e564e669-bf99-42d6-93fa-cec8f1d2406f-ui-tls\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:05.849710 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:05.849445 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e564e669-bf99-42d6-93fa-cec8f1d2406f-feast-data\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:06.028824 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.028740 2562 generic.go:356] "Generic (PLEG): container finished" podID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerID="0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6" exitCode=137 Jun 08 12:11:06.028824 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.028766 2562 generic.go:356] "Generic (PLEG): container finished" podID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerID="7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2" exitCode=137 Jun 08 12:11:06.028824 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.028808 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" Jun 08 12:11:06.029506 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.028820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerDied","Data":"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6"} Jun 08 12:11:06.029506 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.028865 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerDied","Data":"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2"} Jun 08 12:11:06.029506 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.028879 2562 scope.go:117] "RemoveContainer" containerID="35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff" Jun 08 12:11:06.029506 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.028885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5" event={"ID":"e564e669-bf99-42d6-93fa-cec8f1d2406f","Type":"ContainerDied","Data":"42aecffbe7746a02ebc1469b37e25d4c4fcd95301e98b642a2b525bd5b44c143"} Jun 08 12:11:06.031906 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.031880 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" event={"ID":"11fc7f98-97ae-4865-9613-b27126b480d4","Type":"ContainerDied","Data":"aa6fa9c8f000b64ced890a966019fefc5144187c4adf06cb0ec1ae5a150f14cc"} Jun 08 12:11:06.032005 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.031920 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt" Jun 08 12:11:06.039141 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.038773 2562 scope.go:117] "RemoveContainer" containerID="0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6" Jun 08 12:11:06.047475 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.047459 2562 scope.go:117] "RemoveContainer" containerID="ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b" Jun 08 12:11:06.055288 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.055267 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5"] Jun 08 12:11:06.056140 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.056123 2562 scope.go:117] "RemoveContainer" containerID="7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2" Jun 08 12:11:06.060749 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.060726 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-jwvl5"] Jun 08 12:11:06.064354 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.064337 2562 scope.go:117] "RemoveContainer" containerID="493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6" Jun 08 12:11:06.072395 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.072272 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt"] Jun 08 12:11:06.072464 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.072428 2562 scope.go:117] "RemoveContainer" containerID="2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d" Jun 08 12:11:06.078967 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.078942 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-bxxdt"] Jun 08 12:11:06.080114 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.080095 2562 scope.go:117] "RemoveContainer" containerID="35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff" Jun 08 12:11:06.080373 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:06.080355 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff\": container with ID starting with 35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff not found: ID does not exist" containerID="35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff" Jun 08 12:11:06.080420 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.080382 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff"} err="failed to get container status \"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff\": rpc error: code = NotFound desc = could not find container \"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff\": container with ID starting with 35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff not found: ID does not exist" Jun 08 12:11:06.080420 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.080399 2562 scope.go:117] "RemoveContainer" containerID="0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6" Jun 08 12:11:06.080622 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:06.080591 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6\": container with ID starting with 0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6 not found: ID does not exist" containerID="0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6" Jun 08 12:11:06.080692 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.080639 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6"} err="failed to get container status \"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6\": rpc error: code = NotFound desc = could not find container \"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6\": container with ID starting with 0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6 not found: ID does not exist" Jun 08 12:11:06.080692 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.080662 2562 scope.go:117] "RemoveContainer" containerID="ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b" Jun 08 12:11:06.080919 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:06.080900 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b\": container with ID starting with ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b not found: ID does not exist" containerID="ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b" Jun 08 12:11:06.080959 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.080924 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b"} err="failed to get container status \"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b\": rpc error: code = NotFound desc = could not find container \"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b\": container with ID starting with ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b not found: ID does not exist" Jun 08 12:11:06.080959 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.080939 2562 scope.go:117] "RemoveContainer" containerID="7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2" Jun 08 12:11:06.081155 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:06.081137 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2\": container with ID starting with 7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2 not found: ID does not exist" containerID="7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2" Jun 08 12:11:06.081191 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081160 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2"} err="failed to get container status \"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2\": rpc error: code = NotFound desc = could not find container \"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2\": container with ID starting with 7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2 not found: ID does not exist" Jun 08 12:11:06.081191 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081173 2562 scope.go:117] "RemoveContainer" containerID="493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6" Jun 08 12:11:06.081371 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:06.081353 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6\": container with ID starting with 493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6 not found: ID does not exist" containerID="493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6" Jun 08 12:11:06.081440 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081379 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6"} err="failed to get container status \"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6\": rpc error: code = NotFound desc = could not find container \"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6\": container with ID starting with 493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6 not found: ID does not exist" Jun 08 12:11:06.081440 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081397 2562 scope.go:117] "RemoveContainer" containerID="2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d" Jun 08 12:11:06.081666 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:06.081599 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d\": container with ID starting with 2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d not found: ID does not exist" containerID="2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d" Jun 08 12:11:06.081666 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081658 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d"} err="failed to get container status \"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d\": rpc error: code = NotFound desc = could not find container \"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d\": container with ID starting with 2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d not found: ID does not exist" Jun 08 12:11:06.081819 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081677 2562 scope.go:117] "RemoveContainer" containerID="35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff" Jun 08 12:11:06.081923 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081904 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff"} err="failed to get container status \"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff\": rpc error: code = NotFound desc = could not find container \"35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff\": container with ID starting with 35fc4b4dca550ee729c4ab74242f8065947a9ec9d29bd04b6b49d09d73d040ff not found: ID does not exist" Jun 08 12:11:06.081970 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.081925 2562 scope.go:117] "RemoveContainer" containerID="0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6" Jun 08 12:11:06.082150 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082132 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6"} err="failed to get container status \"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6\": rpc error: code = NotFound desc = could not find container \"0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6\": container with ID starting with 0520e0dbb7676b007ac4f7acb301c8ddb3234215348a37e45d8c63803dcd60e6 not found: ID does not exist" Jun 08 12:11:06.082195 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082151 2562 scope.go:117] "RemoveContainer" containerID="ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b" Jun 08 12:11:06.082383 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082358 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b"} err="failed to get container status \"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b\": rpc error: code = NotFound desc = could not find container \"ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b\": container with ID starting with ecdf752049c3c1de1e8b4a01edc6dd7aeab4a3385730b7e20c36f12377c21e6b not found: ID does not exist" Jun 08 12:11:06.082430 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082384 2562 scope.go:117] "RemoveContainer" containerID="7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2" Jun 08 12:11:06.082579 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082565 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2"} err="failed to get container status \"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2\": rpc error: code = NotFound desc = could not find container \"7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2\": container with ID starting with 7d97c1bf47fd551a950ec94d5a980c885e148dfad2f8d128b028c3fb9d928ae2 not found: ID does not exist" Jun 08 12:11:06.082638 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082579 2562 scope.go:117] "RemoveContainer" containerID="493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6" Jun 08 12:11:06.082797 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082780 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6"} err="failed to get container status \"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6\": rpc error: code = NotFound desc = could not find container \"493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6\": container with ID starting with 493eda6561f30662162b2b55e84f383fdea68ba6b1c98104058f5b0874b3c3c6 not found: ID does not exist" Jun 08 12:11:06.082843 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082799 2562 scope.go:117] "RemoveContainer" containerID="2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d" Jun 08 12:11:06.083007 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.082991 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d"} err="failed to get container status \"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d\": rpc error: code = NotFound desc = could not find container \"2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d\": container with ID starting with 2536cffcf15da255535d0e7c5f95ef7211fcd8c1926578525bbd55205b378a6d not found: ID does not exist" Jun 08 12:11:06.083056 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.083007 2562 scope.go:117] "RemoveContainer" containerID="3271fa88b0b1e2bbcd557ab5215c9565d8ef1916547d72bba7545780c8f0abe5" Jun 08 12:11:06.090487 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.090463 2562 scope.go:117] "RemoveContainer" containerID="69ae7205fe3338de545bdb5b81063bf937dc0bf5612078330f549178565f708e" Jun 08 12:11:06.099926 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.099905 2562 scope.go:117] "RemoveContainer" containerID="07ad11610c6ed406ffdff9b060f1cf3558344a7abee469d54b8ccb2d1605fda3" Jun 08 12:11:06.107860 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.107838 2562 scope.go:117] "RemoveContainer" containerID="21eca81b9fe296343617b3b93db36c30474b7d5a08f28e86ced52aa7f8abeb83" Jun 08 12:11:06.115467 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:06.115444 2562 scope.go:117] "RemoveContainer" containerID="213644ef740f5ae0082f9f1a1586bdc4fe363d6a2cbfc6ee3c6d6bde028d6404" Jun 08 12:11:07.202824 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:07.202787 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" path="/var/lib/kubelet/pods/11fc7f98-97ae-4865-9613-b27126b480d4/volumes" Jun 08 12:11:07.203360 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:07.203345 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" path="/var/lib/kubelet/pods/e564e669-bf99-42d6-93fa-cec8f1d2406f/volumes" Jun 08 12:11:17.953999 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:17.953965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:18.082206 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:18.082174 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:23.723668 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:23.723631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:23.787452 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:23.787418 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:24.601026 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.600985 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r2n9n/must-gather-9ppg8"] Jun 08 12:11:24.601300 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601288 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" Jun 08 12:11:24.601343 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601302 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" Jun 08 12:11:24.601343 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601318 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="feast-apply" Jun 08 12:11:24.601343 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601328 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="feast-apply" Jun 08 12:11:24.601343 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601335 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="registry" Jun 08 12:11:24.601343 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601343 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="registry" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601351 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601357 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601366 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="feast-apply" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601371 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="feast-apply" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601376 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="feast-init" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601382 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="feast-init" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601395 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="offline" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601404 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="offline" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601414 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="feast-init" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601420 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="feast-init" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601425 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601430 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601435 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="offline" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601440 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="offline" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601448 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601453 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" Jun 08 12:11:24.601495 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601501 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="offline" Jun 08 12:11:24.602021 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601509 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="online" Jun 08 12:11:24.602021 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601514 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="ui" Jun 08 12:11:24.602021 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601519 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="11fc7f98-97ae-4865-9613-b27126b480d4" containerName="ui" Jun 08 12:11:24.602021 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601525 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="offline" Jun 08 12:11:24.602021 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601530 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="registry" Jun 08 12:11:24.602021 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.601535 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e564e669-bf99-42d6-93fa-cec8f1d2406f" containerName="online" Jun 08 12:11:24.605774 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.605757 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:24.608142 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.608121 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2n9n\"/\"openshift-service-ca.crt\"" Jun 08 12:11:24.608260 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.608182 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2n9n\"/\"kube-root-ca.crt\"" Jun 08 12:11:24.608260 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.608203 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r2n9n\"/\"default-dockercfg-mnnmx\"" Jun 08 12:11:24.623771 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.623742 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2n9n/must-gather-9ppg8"] Jun 08 12:11:24.627138 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.627109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:24.688793 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.688762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:24.711155 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.711123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6mj\" (UniqueName: \"kubernetes.io/projected/f590c5b9-6a0d-4498-901d-301e74ffdf6c-kube-api-access-rp6mj\") pod \"must-gather-9ppg8\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:24.711293 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.711174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f590c5b9-6a0d-4498-901d-301e74ffdf6c-must-gather-output\") pod \"must-gather-9ppg8\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:24.812674 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.812602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6mj\" (UniqueName: \"kubernetes.io/projected/f590c5b9-6a0d-4498-901d-301e74ffdf6c-kube-api-access-rp6mj\") pod \"must-gather-9ppg8\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:24.813033 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.812710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f590c5b9-6a0d-4498-901d-301e74ffdf6c-must-gather-output\") pod \"must-gather-9ppg8\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:24.813033 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.813021 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f590c5b9-6a0d-4498-901d-301e74ffdf6c-must-gather-output\") pod \"must-gather-9ppg8\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:24.822033 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.822001 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6mj\" (UniqueName: \"kubernetes.io/projected/f590c5b9-6a0d-4498-901d-301e74ffdf6c-kube-api-access-rp6mj\") pod \"must-gather-9ppg8\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:24.927415 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:24.927327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:25.058291 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:25.058262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2n9n/must-gather-9ppg8"] Jun 08 12:11:25.060947 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:11:25.060914 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf590c5b9_6a0d_4498_901d_301e74ffdf6c.slice/crio-3c35322ee62e8f10406eda0c1e3bc744c1d3c4241c7991e938320b61f0abbb75 WatchSource:0}: Error finding container 3c35322ee62e8f10406eda0c1e3bc744c1d3c4241c7991e938320b61f0abbb75: Status 404 returned error can't find the container with id 3c35322ee62e8f10406eda0c1e3bc744c1d3c4241c7991e938320b61f0abbb75 Jun 08 12:11:25.062799 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:25.062782 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 08 12:11:25.107329 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:25.107295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" event={"ID":"f590c5b9-6a0d-4498-901d-301e74ffdf6c","Type":"ContainerStarted","Data":"3c35322ee62e8f10406eda0c1e3bc744c1d3c4241c7991e938320b61f0abbb75"} Jun 08 12:11:32.140113 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:32.140065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" event={"ID":"f590c5b9-6a0d-4498-901d-301e74ffdf6c","Type":"ContainerStarted","Data":"b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b"} Jun 08 12:11:32.140113 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:32.140113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" event={"ID":"f590c5b9-6a0d-4498-901d-301e74ffdf6c","Type":"ContainerStarted","Data":"aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230"} Jun 08 12:11:32.156931 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:32.156873 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" podStartSLOduration=2.089290898 podStartE2EDuration="8.156857045s" totalImagesPullingTime="6.067566147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-08 12:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:11:32.155605549 +0000 UTC m=+1951.457382247" watchObservedRunningTime="2026-06-08 12:11:32.156857045 +0000 UTC m=+1951.458633747" Jun 08 12:11:39.481757 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:39.481718 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:41.177854 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:41.177819 2562 generic.go:356] "Generic (PLEG): container finished" podID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerID="aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230" exitCode=0 Jun 08 12:11:41.178305 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:41.177906 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" event={"ID":"f590c5b9-6a0d-4498-901d-301e74ffdf6c","Type":"ContainerDied","Data":"aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230"} Jun 08 12:11:41.178305 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:41.178241 2562 scope.go:117] "RemoveContainer" containerID="aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230" Jun 08 12:11:41.742745 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:41.742708 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r2n9n_must-gather-9ppg8_f590c5b9-6a0d-4498-901d-301e74ffdf6c/gather/0.log" Jun 08 12:11:42.113428 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:42.113397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:42.284872 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:42.284839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:42.384671 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:42.384568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:43.777553 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:43.777522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:43.879301 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:43.879266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:45.148560 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:45.148528 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tk87d_ec330fb8-a79b-4dab-a8c9-5be5d5dcb0ab/global-pull-secret-syncer/0.log" Jun 08 12:11:45.214540 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:45.214511 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" Jun 08 12:11:45.452829 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:45.452749 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-95.ec2.internal_757d7e7b88cdd3b52ce5ab35fd14a29f/haproxy/0.log" Jun 08 12:11:47.091175 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.091132 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-r2n9n/must-gather-9ppg8"] Jun 08 12:11:47.091641 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.091342 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerName="copy" containerID="cri-o://b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b" gracePeriod=2 Jun 08 12:11:47.100822 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.100795 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-r2n9n/must-gather-9ppg8"] Jun 08 12:11:47.102184 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:47.102152 2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f590c5b9-6a0d-4498-901d-301e74ffdf6c\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-r2n9n\"/\"must-gather-9ppg8\": pods \"must-gather-9ppg8\" not found" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:47.320597 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.320574 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r2n9n_must-gather-9ppg8_f590c5b9-6a0d-4498-901d-301e74ffdf6c/copy/0.log" Jun 08 12:11:47.323043 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.323022 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:47.399513 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.399421 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f590c5b9-6a0d-4498-901d-301e74ffdf6c-must-gather-output\") pod \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " Jun 08 12:11:47.399513 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.399503 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp6mj\" (UniqueName: \"kubernetes.io/projected/f590c5b9-6a0d-4498-901d-301e74ffdf6c-kube-api-access-rp6mj\") pod \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\" (UID: \"f590c5b9-6a0d-4498-901d-301e74ffdf6c\") " Jun 08 12:11:47.399870 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.399844 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f590c5b9-6a0d-4498-901d-301e74ffdf6c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f590c5b9-6a0d-4498-901d-301e74ffdf6c" (UID: "f590c5b9-6a0d-4498-901d-301e74ffdf6c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 08 12:11:47.401770 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.401749 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f590c5b9-6a0d-4498-901d-301e74ffdf6c-kube-api-access-rp6mj" (OuterVolumeSpecName: "kube-api-access-rp6mj") pod "f590c5b9-6a0d-4498-901d-301e74ffdf6c" (UID: "f590c5b9-6a0d-4498-901d-301e74ffdf6c"). InnerVolumeSpecName "kube-api-access-rp6mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 08 12:11:47.500927 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.500888 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6mj\" (UniqueName: \"kubernetes.io/projected/f590c5b9-6a0d-4498-901d-301e74ffdf6c-kube-api-access-rp6mj\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:47.500927 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.500921 2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f590c5b9-6a0d-4498-901d-301e74ffdf6c-must-gather-output\") on node \"ip-10-0-137-95.ec2.internal\" DevicePath \"\"" Jun 08 12:11:47.886857 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.886823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:47.962365 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:47.962329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7c57h_2ae4b988-3539-4b07-85ab-690bdfd35c68/konnectivity-agent/0.log" limit=1024 Jun 08 12:11:48.204867 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.204789 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-r2n9n_must-gather-9ppg8_f590c5b9-6a0d-4498-901d-301e74ffdf6c/copy/0.log" Jun 08 12:11:48.205306 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.205127 2562 generic.go:356] "Generic (PLEG): container finished" podID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerID="b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b" exitCode=143 Jun 08 12:11:48.205306 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.205171 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2n9n/must-gather-9ppg8" Jun 08 12:11:48.205306 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.205218 2562 scope.go:117] "RemoveContainer" containerID="b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b" Jun 08 12:11:48.213746 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.213726 2562 scope.go:117] "RemoveContainer" containerID="aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230" Jun 08 12:11:48.224761 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.224737 2562 scope.go:117] "RemoveContainer" containerID="b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b" Jun 08 12:11:48.225051 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:48.225029 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b\": container with ID starting with b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b not found: ID does not exist" containerID="b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b" Jun 08 12:11:48.225101 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.225062 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b"} err="failed to get container status \"b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b\": rpc error: code = NotFound desc = could not find container \"b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b\": container with ID starting with b9a883723d356d9b1687666960eb08f928295a64009a4b859dbe4bdfb69d528b not found: ID does not exist" Jun 08 12:11:48.225101 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.225082 2562 scope.go:117] "RemoveContainer" containerID="aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230" Jun 08 12:11:48.225345 ip-10-0-137-95 kubenswrapper[2562]: E0608 12:11:48.225323 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230\": container with ID starting with aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230 not found: ID does not exist" containerID="aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230" Jun 08 12:11:48.225447 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.225348 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230"} err="failed to get container status \"aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230\": rpc error: code = NotFound desc = could not find container \"aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230\": container with ID starting with aa2483ff9de3022b6b589fc779269dc05dc88fa7652b093b5c530bc9c4099230 not found: ID does not exist" Jun 08 12:11:48.496560 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.496477 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_400f1501-ccfc-4ddb-b98c-7a243ba7e0ad/alertmanager/0.log" Jun 08 12:11:48.567637 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.567590 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_400f1501-ccfc-4ddb-b98c-7a243ba7e0ad/config-reloader/0.log" Jun 08 12:11:48.642343 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.642315 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_400f1501-ccfc-4ddb-b98c-7a243ba7e0ad/kube-rbac-proxy-web/0.log" Jun 08 12:11:48.698256 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.698231 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_400f1501-ccfc-4ddb-b98c-7a243ba7e0ad/kube-rbac-proxy/0.log" Jun 08 12:11:48.760290 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.760180 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_400f1501-ccfc-4ddb-b98c-7a243ba7e0ad/kube-rbac-proxy-metric/0.log" Jun 08 12:11:48.827138 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.827113 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_400f1501-ccfc-4ddb-b98c-7a243ba7e0ad/prom-label-proxy/0.log" Jun 08 12:11:48.869155 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:48.869131 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_400f1501-ccfc-4ddb-b98c-7a243ba7e0ad/init-config-reloader/0.log" Jun 08 12:11:49.109919 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.109891 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-76b68cc874-hcnc8_f9717e3f-f320-4672-87f2-c5fb0afff294/monitoring-plugin/0.log" Jun 08 12:11:49.202865 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.202836 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" path="/var/lib/kubelet/pods/f590c5b9-6a0d-4498-901d-301e74ffdf6c/volumes" Jun 08 12:11:49.349700 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.349670 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zwvdw_348aafc2-831c-480e-9d06-922f732a8754/node-exporter/0.log" Jun 08 12:11:49.381325 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.381247 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zwvdw_348aafc2-831c-480e-9d06-922f732a8754/kube-rbac-proxy/0.log" Jun 08 12:11:49.419317 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.419291 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zwvdw_348aafc2-831c-480e-9d06-922f732a8754/init-textfile/0.log" Jun 08 12:11:49.800100 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.800074 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dd869676c-c59qx_48221b9d-e13e-457c-80d0-0de595ac8293/telemeter-client/0.log" Jun 08 12:11:49.827959 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.827920 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dd869676c-c59qx_48221b9d-e13e-457c-80d0-0de595ac8293/reload/0.log" Jun 08 12:11:49.861444 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:49.861416 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dd869676c-c59qx_48221b9d-e13e-457c-80d0-0de595ac8293/kube-rbac-proxy/0.log" Jun 08 12:11:52.204177 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.204139 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb"] Jun 08 12:11:52.204695 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.204588 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerName="copy" Jun 08 12:11:52.204695 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.204629 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerName="copy" Jun 08 12:11:52.204695 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.204645 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerName="gather" Jun 08 12:11:52.204695 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.204653 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerName="gather" Jun 08 12:11:52.204918 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.204742 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerName="gather" Jun 08 12:11:52.204918 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.204754 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f590c5b9-6a0d-4498-901d-301e74ffdf6c" containerName="copy" Jun 08 12:11:52.209897 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.209876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.212521 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.212493 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2hrr\"/\"openshift-service-ca.crt\"" Jun 08 12:11:52.212644 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.212529 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2hrr\"/\"kube-root-ca.crt\"" Jun 08 12:11:52.213179 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.213164 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b2hrr\"/\"default-dockercfg-87zvt\"" Jun 08 12:11:52.216936 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.216847 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb"] Jun 08 12:11:52.235759 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.235729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-podres\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.235935 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.235764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65q64\" (UniqueName: \"kubernetes.io/projected/54ce910a-cd53-4691-afdd-69dfa18b0ad5-kube-api-access-65q64\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.235935 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.235842 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-lib-modules\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.235935 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.235894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-sys\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.235935 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.235925 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-proc\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337093 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-podres\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337093 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65q64\" (UniqueName: \"kubernetes.io/projected/54ce910a-cd53-4691-afdd-69dfa18b0ad5-kube-api-access-65q64\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337313 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-lib-modules\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337313 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-sys\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337313 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337178 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-proc\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337313 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337222 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-podres\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337313 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337247 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-proc\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337313 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337282 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-sys\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.337313 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.337310 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54ce910a-cd53-4691-afdd-69dfa18b0ad5-lib-modules\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.346248 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.346225 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65q64\" (UniqueName: \"kubernetes.io/projected/54ce910a-cd53-4691-afdd-69dfa18b0ad5-kube-api-access-65q64\") pod \"perf-node-gather-daemonset-s8vmb\" (UID: \"54ce910a-cd53-4691-afdd-69dfa18b0ad5\") " pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.520461 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.520368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:52.649205 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:52.649166 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb"] Jun 08 12:11:52.652706 ip-10-0-137-95 kubenswrapper[2562]: W0608 12:11:52.652669 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod54ce910a_cd53_4691_afdd_69dfa18b0ad5.slice/crio-89794b01852972f2c7284d6555156c91370b977e5cbc8aa25fecaeaba338147f WatchSource:0}: Error finding container 89794b01852972f2c7284d6555156c91370b977e5cbc8aa25fecaeaba338147f: Status 404 returned error can't find the container with id 89794b01852972f2c7284d6555156c91370b977e5cbc8aa25fecaeaba338147f Jun 08 12:11:53.226204 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.226168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" event={"ID":"54ce910a-cd53-4691-afdd-69dfa18b0ad5","Type":"ContainerStarted","Data":"1be9b9d5e10304f15c01c31d768bfba12b843b27967854c18d39222946d74b3c"} Jun 08 12:11:53.226651 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.226218 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" event={"ID":"54ce910a-cd53-4691-afdd-69dfa18b0ad5","Type":"ContainerStarted","Data":"89794b01852972f2c7284d6555156c91370b977e5cbc8aa25fecaeaba338147f"} Jun 08 12:11:53.226651 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.226275 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:53.243225 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.243162 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" podStartSLOduration=1.243147662 podStartE2EDuration="1.243147662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-08 12:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-08 12:11:53.241837876 +0000 UTC m=+1972.543614575" watchObservedRunningTime="2026-06-08 12:11:53.243147662 +0000 UTC m=+1972.544924400" Jun 08 12:11:53.409044 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.408989 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-74ptp_4cad1fdb-2d67-433d-96b9-b2f815229312/dns/0.log" Jun 08 12:11:53.429712 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.429679 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-74ptp_4cad1fdb-2d67-433d-96b9-b2f815229312/kube-rbac-proxy/0.log" Jun 08 12:11:53.579873 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.579847 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qrh9w_96ad9462-64fc-482b-9657-4f5594b57e53/dns-node-resolver/0.log" Jun 08 12:11:53.999445 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:53.999346 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h6rx5_3b88db40-878a-4a55-a597-935a3b17f4fa/node-ca/0.log" Jun 08 12:11:54.688340 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:54.688313 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c75c5578f-n574c_86fee2ac-8922-4a3c-890c-760527f5ea2f/router/0.log" Jun 08 12:11:55.026325 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:55.026237 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7kbj_009f5259-e6a0-45d9-87e3-042f9313cf92/serve-healthcheck-canary/0.log" Jun 08 12:11:55.345031 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:55.344989 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-896549976-4bxmm_a3a0e723-0c3c-4385-a198-ee3b4746fb94/insights-operator/0.log" Jun 08 12:11:55.345595 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:55.345578 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-896549976-4bxmm_a3a0e723-0c3c-4385-a198-ee3b4746fb94/insights-operator/1.log" Jun 08 12:11:55.484388 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:55.484356 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w5k7f_06aa3dd4-5f94-4804-8588-db2b94720738/kube-rbac-proxy/0.log" Jun 08 12:11:55.503146 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:55.503115 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w5k7f_06aa3dd4-5f94-4804-8588-db2b94720738/exporter/0.log" Jun 08 12:11:55.522750 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:55.522714 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w5k7f_06aa3dd4-5f94-4804-8588-db2b94720738/extractor/0.log" Jun 08 12:11:59.240634 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:59.240590 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b2hrr/perf-node-gather-daemonset-s8vmb" Jun 08 12:11:59.639413 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:59.639382 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-955ffdf5f-6s8j9_924ac388-0c8f-42df-aa0a-5f33cd8f5f8d/migrator/0.log" Jun 08 12:11:59.668175 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:11:59.668137 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-955ffdf5f-6s8j9_924ac388-0c8f-42df-aa0a-5f33cd8f5f8d/graceful-termination/0.log" Jun 08 12:12:01.186190 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.186108 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbxx9_4933408b-be9c-45fc-ac36-622024c3d641/kube-multus-additional-cni-plugins/0.log" Jun 08 12:12:01.216448 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.216418 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbxx9_4933408b-be9c-45fc-ac36-622024c3d641/egress-router-binary-copy/0.log" Jun 08 12:12:01.242156 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.242112 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbxx9_4933408b-be9c-45fc-ac36-622024c3d641/cni-plugins/0.log" Jun 08 12:12:01.268332 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.268303 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbxx9_4933408b-be9c-45fc-ac36-622024c3d641/bond-cni-plugin/0.log" Jun 08 12:12:01.293604 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.293581 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbxx9_4933408b-be9c-45fc-ac36-622024c3d641/routeoverride-cni/0.log" Jun 08 12:12:01.323134 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.323106 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbxx9_4933408b-be9c-45fc-ac36-622024c3d641/whereabouts-cni-bincopy/0.log" Jun 08 12:12:01.349155 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.349122 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bbxx9_4933408b-be9c-45fc-ac36-622024c3d641/whereabouts-cni/0.log" Jun 08 12:12:01.691592 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.691561 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s4mjk_a28607fa-dba1-4707-87e6-c4d10f40e16b/kube-multus/0.log" Jun 08 12:12:01.720747 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.720676 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6d2j9_7283a8b3-5f6e-4dc0-addb-cee7c7299122/network-metrics-daemon/0.log" Jun 08 12:12:01.744219 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:01.744192 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6d2j9_7283a8b3-5f6e-4dc0-addb-cee7c7299122/kube-rbac-proxy/0.log" Jun 08 12:12:03.006782 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.006746 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-controller/0.log" Jun 08 12:12:03.024697 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.024670 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/1.log" Jun 08 12:12:03.024878 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.024767 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovn-acl-logging/0.log" Jun 08 12:12:03.045312 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.045284 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/kube-rbac-proxy-node/0.log" Jun 08 12:12:03.067385 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.067308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/kube-rbac-proxy-ovn-metrics/0.log" Jun 08 12:12:03.085239 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.085207 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/northd/0.log" Jun 08 12:12:03.108724 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.108697 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/nbdb/0.log" Jun 08 12:12:03.130085 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.130064 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/sbdb/0.log" Jun 08 12:12:03.176112 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:03.176079 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xr47f_c442b19f-ef6b-49d4-bf2f-e63f4421aab3/ovnkube-controller/0.log" Jun 08 12:12:04.192429 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:04.192394 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hxt4l_8c5da3e0-9dfe-4570-9c0f-2e804419ed32/network-check-target-container/0.log" Jun 08 12:12:05.101338 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:05.101311 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-rjxmg_c86af520-68b1-4f8e-b2cc-23ee36040b53/iptables-alerter/0.log" Jun 08 12:12:05.745419 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:05.745387 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-kn2mj_4f860735-37dd-4726-874c-6ae43f557528/tuned/0.log" Jun 08 12:12:07.333497 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:07.333466 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-854f8b96b6-dpfbw_98c1f2fb-d39c-4829-98aa-cf23f979f58e/cluster-samples-operator/0.log" Jun 08 12:12:07.351734 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:07.351709 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-854f8b96b6-dpfbw_98c1f2fb-d39c-4829-98aa-cf23f979f58e/cluster-samples-operator-watch/0.log" Jun 08 12:12:08.371587 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:08.371556 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-6bc5ff68f5-lhmlx_aecca07d-03c6-4998-947c-a42108f7ac51/service-ca-operator/1.log" Jun 08 12:12:08.372539 ip-10-0-137-95 kubenswrapper[2562]: I0608 12:12:08.372518 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-6bc5ff68f5-lhmlx_aecca07d-03c6-4998-947c-a42108f7ac51/service-ca-operator/0.log"