Jul 28 12:03:06.054247 ip-10-0-134-8 systemd[1]: Starting Kubernetes Kubelet... Jul 28 12:03:06.487463 ip-10-0-134-8 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 12:03:06.487463 ip-10-0-134-8 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 12:03:06.487463 ip-10-0-134-8 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 12:03:06.487463 ip-10-0-134-8 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 12:03:06.487463 ip-10-0-134-8 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 12:03:06.489889 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.489718 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 12:03:06.494862 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494834 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:03:06.494862 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494854 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:03:06.494862 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494860 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:03:06.494862 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494864 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:03:06.494862 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494867 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:03:06.494862 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494870 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494892 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494899 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494902 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494905 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494908 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494911 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494914 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494916 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494919 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494921 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494924 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494926 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494929 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494932 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494934 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494936 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494939 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494941 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494944 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:03:06.495108 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494946 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494948 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494951 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494953 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494956 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494958 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494968 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494971 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494973 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494976 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494979 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494982 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494985 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494987 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494991 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494994 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494997 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.494999 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495002 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495004 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:03:06.495586 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495007 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495009 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495012 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495014 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495017 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495019 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495022 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495024 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495028 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495032 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495035 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495038 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495041 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495044 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495046 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495048 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495051 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495053 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495056 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:03:06.496081 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495058 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495060 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495063 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495066 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495068 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495071 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495073 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495077 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495079 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495082 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495087 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495090 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495093 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495095 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495097 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495100 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495103 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495105 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495108 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:03:06.496554 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495111 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495114 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495116 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495119 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495122 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495124 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495127 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495129 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495132 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495135 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495138 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495140 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495142 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495146 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495148 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495151 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495846 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495852 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495855 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495858 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:03:06.497063 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495861 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495864 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495866 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495869 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495872 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495894 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495897 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495900 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495902 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495905 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495907 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495910 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495913 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495917 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495919 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495922 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495924 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495927 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495930 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495932 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:03:06.497545 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495935 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495937 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495940 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495942 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495944 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495949 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495952 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495956 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495960 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495963 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495986 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495989 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495995 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.495998 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496002 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496004 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496007 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496010 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496013 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:03:06.498069 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496016 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496018 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496021 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496023 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496026 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496029 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496031 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496034 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496037 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496039 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496042 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496044 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496046 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496049 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496051 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496054 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496056 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496059 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496061 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:03:06.498531 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496064 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496066 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496069 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496071 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496074 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496076 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496082 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496085 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496087 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496092 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496095 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496098 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496101 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496103 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496105 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496108 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496110 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496113 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496115 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:03:06.499029 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496118 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496120 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496123 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496126 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496128 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496131 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496134 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496136 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496139 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496141 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496144 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496146 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496148 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496150 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496153 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496155 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496158 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496160 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496562 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496568 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:03:06.499501 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496571 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496574 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496576 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496579 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496582 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496585 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496587 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496590 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496592 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496595 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496597 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496600 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496603 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496605 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496609 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496611 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496614 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496616 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496618 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496621 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:03:06.500021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496624 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496627 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496629 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496632 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496634 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496637 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496639 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496642 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496644 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496646 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496649 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496652 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496654 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496656 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496659 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496663 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496667 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496670 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496673 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496675 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:03:06.500499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496678 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496680 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496683 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496685 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496688 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496690 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496693 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496696 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496699 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496702 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496704 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496707 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496709 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496712 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496714 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496717 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496719 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496722 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496724 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496726 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:03:06.500988 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496729 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496731 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496734 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496736 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496739 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496741 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496744 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496746 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496749 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496751 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496755 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496757 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496760 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496762 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496764 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496768 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496771 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496775 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496777 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:03:06.501499 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496784 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496787 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496790 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496794 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496798 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496801 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496804 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496806 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496809 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496812 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496814 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496816 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496819 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496822 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496824 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496827 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496830 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.496833 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496938 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496947 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 12:03:06.502018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496954 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496958 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496962 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496966 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496970 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496974 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496977 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496980 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496984 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496987 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496990 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496993 2564 flags.go:64] FLAG: --cgroup-root="" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.496998 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497001 2564 flags.go:64] FLAG: --client-ca-file="" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497005 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497007 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497016 2564 flags.go:64] FLAG: --cluster-domain="" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497019 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497022 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497025 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497029 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497033 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497035 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 12:03:06.502530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497039 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497042 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497045 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497048 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497050 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497053 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497056 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497060 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497063 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497066 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497069 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497072 2564 flags.go:64] FLAG: --enable-server="true" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497076 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497080 2564 flags.go:64] FLAG: --event-burst="100" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497083 2564 flags.go:64] FLAG: --event-qps="50" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497086 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497089 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497092 2564 flags.go:64] FLAG: --eviction-hard="" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497095 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497098 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497101 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497105 2564 flags.go:64] FLAG: --eviction-soft="" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497108 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497111 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497114 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 12:03:06.503160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497117 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497120 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497122 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497125 2564 flags.go:64] FLAG: --feature-gates="" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497128 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497131 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497134 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497137 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497140 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497143 2564 flags.go:64] FLAG: --help="false" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497145 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497148 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497151 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497154 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497157 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497160 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497163 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497165 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497168 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497172 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497175 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497177 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497180 2564 flags.go:64] FLAG: --kube-reserved="" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497183 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 12:03:06.503789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497185 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497188 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497191 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497193 2564 flags.go:64] FLAG: --lock-file="" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497198 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497201 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497204 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497209 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497213 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497216 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497219 2564 flags.go:64] FLAG: --logging-format="text" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497222 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497225 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497227 2564 flags.go:64] FLAG: --manifest-url="" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497230 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497234 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497237 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497241 2564 flags.go:64] FLAG: --max-pods="110" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497244 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497246 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497249 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497252 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497255 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497258 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497261 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 12:03:06.504414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497268 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497270 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497275 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497278 2564 flags.go:64] FLAG: --pod-cidr="" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497282 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497287 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497290 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497293 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497296 2564 flags.go:64] FLAG: --port="10250" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497298 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497301 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0707f12fcae03ed36" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497305 2564 flags.go:64] FLAG: --qos-reserved="" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497309 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497311 2564 flags.go:64] FLAG: --register-node="true" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497314 2564 flags.go:64] FLAG: --register-with-taints="" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497318 2564 flags.go:64] FLAG: --registry-burst="10" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497321 2564 flags.go:64] FLAG: --registry-qps="5" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497324 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497327 2564 flags.go:64] FLAG: --reserved-memory="" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497331 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497333 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497336 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497339 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497341 2564 flags.go:64] FLAG: --runonce="false" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497344 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 12:03:06.505048 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497347 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497349 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497352 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497355 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497357 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497360 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497363 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497366 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497368 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497372 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497375 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497378 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497380 2564 flags.go:64] FLAG: --system-cgroups="" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497383 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497388 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497391 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497393 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497398 2564 flags.go:64] FLAG: --tls-min-version="" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497400 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497404 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497407 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497409 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497412 2564 flags.go:64] FLAG: --v="2" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497416 2564 flags.go:64] FLAG: --version="false" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497420 2564 flags.go:64] FLAG: --vmodule="" Jul 28 12:03:06.505654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497423 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.497426 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497563 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497568 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497571 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497574 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497577 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497579 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497582 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497584 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497587 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497589 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497592 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497594 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497596 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497599 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497603 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497605 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497608 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497610 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:03:06.506274 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497614 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497616 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497619 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497621 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497624 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497626 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497630 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497632 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497635 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497637 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497640 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497642 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497645 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497648 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497650 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497653 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497655 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497658 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497660 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497663 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:03:06.506800 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497665 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497667 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497671 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497675 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497678 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497680 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497684 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497688 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497692 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497695 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497697 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497700 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497702 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497705 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497707 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497710 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497712 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497714 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:03:06.507858 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497720 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497723 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497726 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497729 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497731 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497734 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497736 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497739 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497742 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497744 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497746 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497749 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497751 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497754 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497756 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497759 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497761 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497764 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497766 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497769 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:03:06.508319 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497771 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497774 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497777 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497780 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497782 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497785 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497787 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497790 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497792 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497794 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497797 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497799 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497803 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497806 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497808 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497810 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497813 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497815 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497818 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497821 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:03:06.508786 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497829 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497832 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.497834 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.498735 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.505501 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.505524 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.506338 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.506347 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506405 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506411 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506415 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506418 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506421 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506424 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:03:06.509291 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506427 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506429 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506432 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506434 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506437 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506440 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506442 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506445 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506447 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506450 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506452 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506455 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506457 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506459 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506462 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506464 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506467 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506470 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506472 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506475 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:03:06.509636 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506477 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506480 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506483 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506486 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506489 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506491 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506494 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506496 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506499 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506502 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506504 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506507 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506510 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506513 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506515 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506518 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506520 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506522 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506525 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506527 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:03:06.510146 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506530 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506532 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506535 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506537 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506541 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506545 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506547 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506550 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506553 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506556 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506558 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506561 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506563 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506566 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506569 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506572 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506575 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506577 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506580 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:03:06.510677 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506582 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506585 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506587 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506591 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506594 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506597 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506600 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506602 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506605 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506608 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506610 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506613 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506615 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506617 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506620 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506622 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506625 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506627 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506629 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:03:06.511175 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506632 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506635 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506637 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506640 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506642 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506645 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506648 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506650 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506653 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506656 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506666 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506670 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506672 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506675 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506678 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:03:06.511628 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.506683 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506799 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506804 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506807 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506811 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506814 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506817 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506820 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506822 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506825 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506829 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506831 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506834 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506837 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506840 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506842 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506845 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506847 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506850 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506852 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 12:03:06.512064 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506854 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506857 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506859 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506862 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506865 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506867 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506870 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506872 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506889 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506892 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506895 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506897 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506900 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506902 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506905 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506907 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506910 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506912 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506915 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506917 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 12:03:06.512533 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506919 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506922 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506924 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506928 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506932 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506935 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506937 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506939 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506942 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506944 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506947 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506949 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506952 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506954 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506957 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506959 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506962 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506965 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506967 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506970 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 12:03:06.513038 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506972 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506975 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506978 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506980 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506982 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506985 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506987 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506990 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506992 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506994 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.506998 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507001 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507004 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507007 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507010 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507028 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507032 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507035 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507038 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 12:03:06.513523 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507040 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507043 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507046 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507050 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507053 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507056 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507059 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507062 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507064 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507067 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507069 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507072 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507075 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507078 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507080 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507083 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507086 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507088 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507091 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 12:03:06.514101 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507093 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:06.507096 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.507100 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.507208 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.509204 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.510439 2564 server.go:1019] "Starting client certificate rotation" Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.510541 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.510575 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 12:03:06.514710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.512992 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 12:03:06.517584 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.517563 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 12:03:06.526743 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.526725 2564 log.go:25] "Validated CRI v1 runtime API" Jul 28 12:03:06.528610 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.528596 2564 log.go:25] "Validated CRI v1 image API" Jul 28 12:03:06.529364 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.529345 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 12:03:06.531775 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.531755 2564 fs.go:135] Filesystem UUIDs: map[22e09205-0e98-4cfc-b2a5-568f2c6d18e7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a4f35ecb-b6db-48d1-9aef-014f1a881229:/dev/nvme0n1p3] Jul 28 12:03:06.531827 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.531776 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 12:03:06.537581 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.537482 2564 manager.go:216] Machine: {Timestamp:2026-07-28 12:03:06.536034367 +0000 UTC m=+0.352964175 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100878 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0b9cded720181b41c6feffb685e0 SystemUUID:ec2e0b9c-ded7-2018-1b41-c6feffb685e0 BootID:78ac8bdf-173b-407e-8f96-209110bd9b69 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:71:00:28:85:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:71:00:28:85:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:c2:8d:a3:5a:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 12:03:06.537581 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.537578 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 12:03:06.537692 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.537647 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 12:03:06.540795 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.540768 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 12:03:06.540942 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.540798 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 12:03:06.542041 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.542029 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 12:03:06.542087 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.542045 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 12:03:06.542087 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.542053 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 12:03:06.542826 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.542807 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 12:03:06.543625 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.543615 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 12:03:06.545261 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545245 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 12:03:06.545470 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545459 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 28 12:03:06.545594 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545584 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 12:03:06.545644 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545614 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 28 12:03:06.545644 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545625 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 12:03:06.545644 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545641 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 12:03:06.545771 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545649 2564 kubelet.go:398] "Adding apiserver pod source" Jul 28 12:03:06.545771 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.545658 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 12:03:06.546192 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.546172 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 12:03:06.546274 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.546204 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 12:03:06.549992 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.549960 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 12:03:06.552027 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.552007 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 12:03:06.552129 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.552042 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 12:03:06.553728 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553714 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 12:03:06.553772 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553732 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 12:03:06.553772 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553740 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 12:03:06.553772 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553748 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 12:03:06.553772 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553761 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 12:03:06.553772 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553768 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 12:03:06.553932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553778 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 12:03:06.553932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553783 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 12:03:06.553932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553790 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 12:03:06.553932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553796 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 12:03:06.553932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553805 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 12:03:06.553932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.553810 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 12:03:06.554964 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.554951 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 12:03:06.555003 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.554967 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 12:03:06.558312 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.558299 2564 server.go:1268] "Started kubelet" Jul 28 12:03:06.558437 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.558410 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 12:03:06.558526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.558393 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 12:03:06.558594 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.558544 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 12:03:06.559313 ip-10-0-134-8 systemd[1]: Started Kubernetes Kubelet. Jul 28 12:03:06.560053 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.560036 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 12:03:06.560053 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.560038 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 12:03:06.560266 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.560200 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 12:03:06.560323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.560288 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 12:03:06.561159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.561147 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 28 12:03:06.563869 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.563851 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 12:03:06.566327 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.565366 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727db68ec31e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.55826819 +0000 UTC m=+0.375197997,LastTimestamp:2026-07-28 12:03:06.55826819 +0000 UTC m=+0.375197997,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.567209 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.567196 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 12:03:06.567276 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.567251 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 12:03:06.567400 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.567279 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:06.567833 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.567798 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 12:03:06.567928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.567919 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 12:03:06.567981 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.567935 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 12:03:06.568054 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568042 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 28 12:03:06.568098 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568055 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 28 12:03:06.568509 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568494 2564 factory.go:153] Registering CRI-O factory Jul 28 12:03:06.568575 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568512 2564 factory.go:223] Registration of the crio container factory successfully Jul 28 12:03:06.568575 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568565 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 12:03:06.568575 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568572 2564 factory.go:55] Registering systemd factory Jul 28 12:03:06.568668 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568578 2564 factory.go:223] Registration of the systemd container factory successfully Jul 28 12:03:06.568668 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568596 2564 factory.go:103] Registering Raw factory Jul 28 12:03:06.568668 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568608 2564 manager.go:1196] Started watching for new ooms in manager Jul 28 12:03:06.568968 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.568955 2564 manager.go:349] Starting recovery of all containers Jul 28 12:03:06.572124 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.572077 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 12:03:06.572237 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.572214 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 12:03:06.581272 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.581253 2564 manager.go:354] Recovery completed Jul 28 12:03:06.582634 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.582602 2564 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 28 12:03:06.585593 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.585580 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:06.587386 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.587372 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:06.587456 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.587399 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:06.587456 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.587409 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:06.587899 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.587869 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 12:03:06.587955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.587900 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 12:03:06.587955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.587890 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:06.587955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.587915 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 28 12:03:06.588627 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.588612 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:06.588692 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.588642 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:06.588692 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.588656 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:06.589769 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.589754 2564 policy_none.go:49] "None policy: Start" Jul 28 12:03:06.589866 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.589783 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 12:03:06.589866 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.589794 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 12:03:06.590017 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.589936 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727db84b0fb2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.587385778 +0000 UTC m=+0.404315585,LastTimestamp:2026-07-28 12:03:06.587385778 +0000 UTC m=+0.404315585,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.591072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.591060 2564 policy_none.go:47] "Start" Jul 28 12:03:06.602560 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.602500 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727db84b55bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.587403711 +0000 UTC m=+0.404333518,LastTimestamp:2026-07-28 12:03:06.587403711 +0000 UTC m=+0.404333518,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.616143 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.615998 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727db84b7857 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.587412567 +0000 UTC m=+0.404342375,LastTimestamp:2026-07-28 12:03:06.587412567 +0000 UTC m=+0.404342375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.627454 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.627440 2564 manager.go:337] "Starting Device Plugin manager" Jul 28 12:03:06.627524 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.627475 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 12:03:06.627524 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.627484 2564 server.go:87] "Starting device plugin registration server" Jul 28 12:03:06.627717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.627705 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 12:03:06.627756 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.627720 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 12:03:06.627816 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.627801 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 12:03:06.627892 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.627865 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 12:03:06.627892 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.627891 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 12:03:06.628352 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.628333 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 12:03:06.628413 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.628391 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:06.629353 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.629299 2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-8.ec2.internal.18c6727db84b0fb2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727db84b0fb2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-8.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.587385778 +0000 UTC m=+0.404315585,LastTimestamp:2026-07-28 12:03:06.588627145 +0000 UTC m=+0.405556956,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.641220 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.641158 2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-8.ec2.internal.18c6727db84b55bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727db84b55bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-8.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.587403711 +0000 UTC m=+0.404333518,LastTimestamp:2026-07-28 12:03:06.588649057 +0000 UTC m=+0.405578866,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.656917 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.656815 2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-8.ec2.internal.18c6727db84b7857\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727db84b7857 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-8.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.587412567 +0000 UTC m=+0.404342375,LastTimestamp:2026-07-28 12:03:06.588661876 +0000 UTC m=+0.405591686,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.666370 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.666338 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hc46" Jul 28 12:03:06.668116 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.668050 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c6727dbace55dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-28 12:03:06.629543389 +0000 UTC m=+0.446473187,LastTimestamp:2026-07-28 12:03:06.629543389 +0000 UTC m=+0.446473187,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}" Jul 28 12:03:06.685473 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.685449 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4hc46" Jul 28 12:03:06.728085 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.728066 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:06.729501 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.728941 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:06.729501 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.728969 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:06.729501 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.728979 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:06.729501 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.729002 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.749993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.749908 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.749993 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.749942 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-8.ec2.internal\": node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:06.772826 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.772791 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 12:03:06.774260 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.774244 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 12:03:06.774332 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.774288 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 12:03:06.774332 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.774313 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 12:03:06.774423 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.774354 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 12:03:06.780570 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.780556 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:03:06.797666 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.797650 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:06.875061 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.875003 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"] Jul 28 12:03:06.875216 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.875108 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:06.877789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.877763 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:06.877951 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.877808 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:06.877951 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.877822 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:06.879060 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879043 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:06.879157 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879145 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.879202 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879185 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:06.879835 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879818 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:06.879932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879843 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:06.879932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879871 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:06.879932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879909 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:06.880033 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879935 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:06.880033 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.879946 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:06.881192 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.881178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.881252 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.881201 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:06.881791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.881777 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:06.881871 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.881802 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:06.881871 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.881811 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:06.898003 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.897984 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:06.907411 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.907393 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-8.ec2.internal\" not found" node="ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.911690 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.911676 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-8.ec2.internal\" not found" node="ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.970307 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.970273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9adaaed0eb9e1e379ca9054dcfb08b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"c9adaaed0eb9e1e379ca9054dcfb08b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.970307 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.970306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9adaaed0eb9e1e379ca9054dcfb08b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"c9adaaed0eb9e1e379ca9054dcfb08b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.970450 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:06.970327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b4c93f4e70558b77ddac96ac849afbc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-8.ec2.internal\" (UID: \"5b4c93f4e70558b77ddac96ac849afbc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" Jul 28 12:03:06.998215 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:06.998195 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.071144 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.071066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9adaaed0eb9e1e379ca9054dcfb08b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"c9adaaed0eb9e1e379ca9054dcfb08b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.071144 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.071099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9adaaed0eb9e1e379ca9054dcfb08b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"c9adaaed0eb9e1e379ca9054dcfb08b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.071144 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.071114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b4c93f4e70558b77ddac96ac849afbc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-8.ec2.internal\" (UID: \"5b4c93f4e70558b77ddac96ac849afbc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.071144 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.071143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b4c93f4e70558b77ddac96ac849afbc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-8.ec2.internal\" (UID: \"5b4c93f4e70558b77ddac96ac849afbc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.071338 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.071159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c9adaaed0eb9e1e379ca9054dcfb08b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"c9adaaed0eb9e1e379ca9054dcfb08b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.071338 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.071173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9adaaed0eb9e1e379ca9054dcfb08b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"c9adaaed0eb9e1e379ca9054dcfb08b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.098420 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.098400 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.199005 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.198964 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.211144 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.211122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.215543 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.215126 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" Jul 28 12:03:07.299953 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.299918 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.400445 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.400372 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.500980 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.500931 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.510083 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.510065 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 12:03:07.510260 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.510242 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 12:03:07.601577 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.601546 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.668011 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.667925 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 12:03:07.668986 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.668964 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory" Jul 28 12:03:07.669095 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.668999 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 12:03:07.669095 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.669009 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID" Jul 28 12:03:07.692433 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.692404 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 11:58:07 +0000 UTC" deadline="2027-12-22 10:27:25.403636735 +0000 UTC" Jul 28 12:03:07.692433 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.692429 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12286h24m17.711211582s" Jul 28 12:03:07.702621 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.702546 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.746167 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.746147 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:03:07.803078 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.803048 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.840788 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:07.840761 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9adaaed0eb9e1e379ca9054dcfb08b6.slice/crio-75d4d801fcc7b8104065e87a5e1a2329f06dbcda1c14e0e040a0183f9fce3ea4 WatchSource:0}: Error finding container 75d4d801fcc7b8104065e87a5e1a2329f06dbcda1c14e0e040a0183f9fce3ea4: Status 404 returned error can't find the container with id 75d4d801fcc7b8104065e87a5e1a2329f06dbcda1c14e0e040a0183f9fce3ea4 Jul 28 12:03:07.840993 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:07.840971 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4c93f4e70558b77ddac96ac849afbc.slice/crio-1b74d61a6dc4fc1aa76c46fe3bbaf921cca1eae7d35de120139d616d98c938d1 WatchSource:0}: Error finding container 1b74d61a6dc4fc1aa76c46fe3bbaf921cca1eae7d35de120139d616d98c938d1: Status 404 returned error can't find the container with id 1b74d61a6dc4fc1aa76c46fe3bbaf921cca1eae7d35de120139d616d98c938d1 Jul 28 12:03:07.845553 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.845538 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:03:07.903304 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:07.903265 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found" Jul 28 12:03:07.983732 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:07.983671 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:03:08.068254 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.068219 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" Jul 28 12:03:08.083731 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.083703 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 12:03:08.085399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.085378 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" Jul 28 12:03:08.105253 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.105233 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 12:03:08.137826 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.137804 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:03:08.547093 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.547051 2564 apiserver.go:52] "Watching apiserver" Jul 28 12:03:08.555008 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.554975 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 12:03:08.557619 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.557581 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal","openshift-multus/multus-wb5fb","openshift-multus/network-metrics-daemon-gklcw","openshift-network-diagnostics/network-check-target-hmlhv","openshift-network-operator/iptables-alerter-ndsld","kube-system/global-pull-secret-syncer-4mk2d","kube-system/konnectivity-agent-wr2z4","kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal","openshift-cluster-node-tuning-operator/tuned-82vhr","openshift-multus/multus-additional-cni-plugins-wnjtb","openshift-ovn-kubernetes/ovnkube-node-z7967","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz","openshift-dns/node-resolver-w85q8","openshift-image-registry/node-ca-sh2lt"] Jul 28 12:03:08.559496 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.559469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.560951 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.560925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.561875 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.561853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:08.562004 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.561946 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:08.562872 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.562853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-52cx5\"" Jul 28 12:03:08.563006 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.562944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 12:03:08.563075 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.562970 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 12:03:08.563556 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.563534 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:08.563671 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.563608 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:08.564569 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.564502 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 12:03:08.564796 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.564777 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzmkb\"" Jul 28 12:03:08.565215 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.565188 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 12:03:08.570107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.568941 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 12:03:08.572223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.571529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.573668 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.573558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.573668 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.573621 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:08.576917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.574025 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.576917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.574828 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 12:03:08.576917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.575067 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 12:03:08.576917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.575381 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trx7v\"" Jul 28 12:03:08.576917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.575599 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 12:03:08.576917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.575806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.582911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.582881 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 12:03:08.583355 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.583337 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 12:03:08.583581 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.583566 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 12:03:08.583923 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.583888 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7n2s\"" Jul 28 12:03:08.584319 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.584301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.584466 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.584452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.586111 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:08.586240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-host-slash\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.586240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586134 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-psp56\"" Jul 28 12:03:08.586240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586163 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 12:03:08.586446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2knm\" (UniqueName: \"kubernetes.io/projected/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-kube-api-access-j2knm\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.586446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d522fb7-96d0-40b8-aa15-33ca968f946c-agent-certs\") pod \"konnectivity-agent-wr2z4\" (UID: \"5d522fb7-96d0-40b8-aa15-33ca968f946c\") " pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.586446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ea75a61-bc22-494b-8843-902854593d6c-hosts-file\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.586446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5z25\" (UniqueName: \"kubernetes.io/projected/1ea75a61-bc22-494b-8843-902854593d6c-kube-api-access-z5z25\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.586446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586439 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 12:03:08.586732 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55wp5\" (UniqueName: \"kubernetes.io/projected/0afc5b7d-6f03-4e22-b639-88fa243b94d3-kube-api-access-55wp5\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:08.586732 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-iptables-alerter-script\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.586732 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d522fb7-96d0-40b8-aa15-33ca968f946c-konnectivity-ca\") pod \"konnectivity-agent-wr2z4\" (UID: \"5d522fb7-96d0-40b8-aa15-33ca968f946c\") " pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.586732 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea75a61-bc22-494b-8843-902854593d6c-tmp-dir\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.586732 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.586597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:08.587658 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.587636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 12:03:08.588214 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.588195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.590388 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.590280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.591281 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.590855 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 12:03:08.591281 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.590865 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q4t5c\"" Jul 28 12:03:08.591281 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fxtvl\"" Jul 28 12:03:08.591447 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591308 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 12:03:08.591447 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mdlgc\"" Jul 28 12:03:08.591691 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591662 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 12:03:08.591756 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591723 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 12:03:08.591811 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 12:03:08.591918 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591889 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 12:03:08.591994 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.591948 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 12:03:08.592262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.592240 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 12:03:08.592455 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.592438 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 12:03:08.592655 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.592638 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 12:03:08.593518 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.593500 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 12:03:08.593727 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.593499 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 12:03:08.594103 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.594085 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 12:03:08.594298 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.594275 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2chr9\"" Jul 28 12:03:08.595015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.594996 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 12:03:08.596867 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.596845 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 12:03:08.629879 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.629851 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rp97" Jul 28 12:03:08.640916 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.640889 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rp97" Jul 28 12:03:08.668709 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.668690 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 12:03:08.687793 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-var-lib-kubelet\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.687930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-kubelet\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.687930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-ovn\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.687930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.687930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-multus-certs\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.687930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-host-slash\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d522fb7-96d0-40b8-aa15-33ca968f946c-agent-certs\") pod \"konnectivity-agent-wr2z4\" (UID: \"5d522fb7-96d0-40b8-aa15-33ca968f946c\") " pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.687982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5z25\" (UniqueName: \"kubernetes.io/projected/1ea75a61-bc22-494b-8843-902854593d6c-kube-api-access-z5z25\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55wp5\" (UniqueName: \"kubernetes.io/projected/0afc5b7d-6f03-4e22-b639-88fa243b94d3-kube-api-access-55wp5\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-tuned\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e57a037-dd6f-4793-bfa6-6145355fa80f-serviceca\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-k8s-cni-cncf-io\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-kubelet\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d522fb7-96d0-40b8-aa15-33ca968f946c-konnectivity-ca\") pod \"konnectivity-agent-wr2z4\" (UID: \"5d522fb7-96d0-40b8-aa15-33ca968f946c\") " pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:08.688195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/caf368fa-46b2-4bbc-a2a0-d3f35022719a-dbus\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gn6k\" (UniqueName: \"kubernetes.io/projected/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-kube-api-access-8gn6k\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-slash\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-etc-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjn4n\" (UniqueName: \"kubernetes.io/projected/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-kube-api-access-kjn4n\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-netns\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-host-slash\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-sys-fs\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e57a037-dd6f-4793-bfa6-6145355fa80f-host\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-etc-kubernetes\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-node-log\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovn-node-metrics-cert\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.688703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-systemd\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688802 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.688935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d522fb7-96d0-40b8-aa15-33ca968f946c-konnectivity-ca\") pod \"konnectivity-agent-wr2z4\" (UID: \"5d522fb7-96d0-40b8-aa15-33ca968f946c\") " pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.689093 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.689178 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs podName:0afc5b7d-6f03-4e22-b639-88fa243b94d3 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:09.189143978 +0000 UTC m=+3.006073804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs") pod "network-metrics-daemon-gklcw" (UID: "0afc5b7d-6f03-4e22-b639-88fa243b94d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-lib-modules\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovnkube-script-lib\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689266 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cnibin\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-os-release\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.689366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2knm\" (UniqueName: \"kubernetes.io/projected/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-kube-api-access-j2knm\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysctl-conf\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-run\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-cni-binary-copy\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-socket-dir-parent\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-cni-bin\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-device-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-tmp\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-cni-netd\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-system-cni-dir\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpc8t\" (UniqueName: \"kubernetes.io/projected/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-kube-api-access-rpc8t\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-sys\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-cni-bin\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.690066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-log-socket\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl8tl\" (UniqueName: \"kubernetes.io/projected/6abe718b-42b3-4ae8-8051-8bb46abd4db4-kube-api-access-kl8tl\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.689998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-daemon-config\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-hostroot\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-var-lib-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-env-overrides\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-iptables-alerter-script\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea75a61-bc22-494b-8843-902854593d6c-tmp-dir\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlstf\" (UniqueName: \"kubernetes.io/projected/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-kube-api-access-zlstf\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovnkube-config\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-socket-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-kubernetes\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6gm\" (UniqueName: \"kubernetes.io/projected/4e57a037-dd6f-4793-bfa6-6145355fa80f-kube-api-access-2n6gm\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-cnibin\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.690824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/caf368fa-46b2-4bbc-a2a0-d3f35022719a-kubelet-config\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-system-cni-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-cni-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-systemd-units\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-run-netns\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-iptables-alerter-script\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.690733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysctl-d\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea75a61-bc22-494b-8843-902854593d6c-tmp-dir\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ea75a61-bc22-494b-8843-902854593d6c-hosts-file\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-modprobe-d\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-host\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-os-release\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-cni-multus\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-conf-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-registration-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.691717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysconfig\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.692581 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-systemd\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.692581 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.691673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ea75a61-bc22-494b-8843-902854593d6c-hosts-file\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.693706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.693684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d522fb7-96d0-40b8-aa15-33ca968f946c-agent-certs\") pod \"konnectivity-agent-wr2z4\" (UID: \"5d522fb7-96d0-40b8-aa15-33ca968f946c\") " pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.702035 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.701636 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:03:08.702035 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.701660 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:03:08.702035 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.701675 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bqxhc for pod openshift-network-diagnostics/network-check-target-hmlhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:08.702035 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.701733 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc podName:12686a8e-d528-4138-a8ff-0eaf60244bd2 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:09.201715129 +0000 UTC m=+3.018644943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bqxhc" (UniqueName: "kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc") pod "network-check-target-hmlhv" (UID: "12686a8e-d528-4138-a8ff-0eaf60244bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:08.705189 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.705165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55wp5\" (UniqueName: \"kubernetes.io/projected/0afc5b7d-6f03-4e22-b639-88fa243b94d3-kube-api-access-55wp5\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:08.705642 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.705620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5z25\" (UniqueName: \"kubernetes.io/projected/1ea75a61-bc22-494b-8843-902854593d6c-kube-api-access-z5z25\") pod \"node-resolver-w85q8\" (UID: \"1ea75a61-bc22-494b-8843-902854593d6c\") " pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.707599 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.707581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2knm\" (UniqueName: \"kubernetes.io/projected/dcf5bafc-60f8-4a8e-964c-b697c819f0f4-kube-api-access-j2knm\") pod \"iptables-alerter-ndsld\" (UID: \"dcf5bafc-60f8-4a8e-964c-b697c819f0f4\") " pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.740996 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.740959 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 12:03:08.784123 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.783984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" event={"ID":"c9adaaed0eb9e1e379ca9054dcfb08b6","Type":"ContainerStarted","Data":"75d4d801fcc7b8104065e87a5e1a2329f06dbcda1c14e0e040a0183f9fce3ea4"} Jul 28 12:03:08.786620 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.786592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" event={"ID":"5b4c93f4e70558b77ddac96ac849afbc","Type":"ContainerStarted","Data":"1b74d61a6dc4fc1aa76c46fe3bbaf921cca1eae7d35de120139d616d98c938d1"} Jul 28 12:03:08.792416 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpc8t\" (UniqueName: \"kubernetes.io/projected/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-kube-api-access-rpc8t\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.792561 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-sys\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.792561 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-cni-bin\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.792561 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-log-socket\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.792561 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl8tl\" (UniqueName: \"kubernetes.io/projected/6abe718b-42b3-4ae8-8051-8bb46abd4db4-kube-api-access-kl8tl\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.792561 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-daemon-config\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-hostroot\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-var-lib-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-env-overrides\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlstf\" (UniqueName: \"kubernetes.io/projected/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-kube-api-access-zlstf\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovnkube-config\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-socket-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-kubernetes\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6gm\" (UniqueName: \"kubernetes.io/projected/4e57a037-dd6f-4793-bfa6-6145355fa80f-kube-api-access-2n6gm\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-cnibin\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.792834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/caf368fa-46b2-4bbc-a2a0-d3f35022719a-kubelet-config\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-system-cni-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-cni-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.792972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-systemd-units\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-run-netns\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysctl-d\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-modprobe-d\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-host\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-os-release\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-cni-multus\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-conf-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-registration-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysconfig\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-systemd\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-var-lib-kubelet\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-kubelet\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.793478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-ovn\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-daemon-config\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-sys\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-hostroot\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-var-lib-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-multus-certs\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-tuned\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e57a037-dd6f-4793-bfa6-6145355fa80f-serviceca\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-k8s-cni-cncf-io\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-kubelet\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/caf368fa-46b2-4bbc-a2a0-d3f35022719a-dbus\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gn6k\" (UniqueName: \"kubernetes.io/projected/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-kube-api-access-8gn6k\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-slash\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-etc-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjn4n\" (UniqueName: \"kubernetes.io/projected/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-kube-api-access-kjn4n\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-netns\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.794330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-sys-fs\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e57a037-dd6f-4793-bfa6-6145355fa80f-host\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-etc-kubernetes\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-node-log\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-env-overrides\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovn-node-metrics-cert\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-systemd\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.793999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-lib-modules\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovnkube-script-lib\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cnibin\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-os-release\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysctl-conf\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-run\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.795234 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-cni-binary-copy\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-socket-dir-parent\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-cni-bin\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-device-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-tmp\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-cni-netd\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-system-cni-dir\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-cni-bin\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-log-socket\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794730 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovnkube-config\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-socket-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-kubernetes\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.794946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-cnibin\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.795432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.796159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.795490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-host\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.795547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/caf368fa-46b2-4bbc-a2a0-d3f35022719a-kubelet-config\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.795624 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:08.795688 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret podName:caf368fa-46b2-4bbc-a2a0-d3f35022719a nodeName:}" failed. No retries permitted until 2026-07-28 12:03:09.295670543 +0000 UTC m=+3.112600348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret") pod "global-pull-secret-syncer-4mk2d" (UID: "caf368fa-46b2-4bbc-a2a0-d3f35022719a") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.795686 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-multus-certs\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.795883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-system-cni-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-os-release\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysctl-d\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796202 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-systemd-units\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-cni-multus\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-run-netns\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-conf-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-modprobe-d\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796371 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-k8s-cni-cncf-io\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-var-lib-kubelet\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/caf368fa-46b2-4bbc-a2a0-d3f35022719a-dbus\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.796809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-slash\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-etc-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796791 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4e57a037-dd6f-4793-bfa6-6145355fa80f-serviceca\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-registration-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-run\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-systemd\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysconfig\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-systemd\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-var-lib-kubelet\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.796984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-kubelet\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797029 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-os-release\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-ovn\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-lib-modules\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-host-run-netns\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-sys-fs\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cnibin\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-node-log\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.797713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-etc-kubernetes\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e57a037-dd6f-4793-bfa6-6145355fa80f-host\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-socket-dir-parent\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-run-openvswitch\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-sysctl-conf\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-cni-netd\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-host-cni-bin\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-device-dir\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6abe718b-42b3-4ae8-8051-8bb46abd4db4-system-cni-dir\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-cni-binary-copy\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6abe718b-42b3-4ae8-8051-8bb46abd4db4-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-multus-cni-dir\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.798527 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.797887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovnkube-script-lib\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.801788 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.801741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-etc-tuned\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.805049 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.805004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-tmp\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.811418 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.811286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpc8t\" (UniqueName: \"kubernetes.io/projected/6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b-kube-api-access-rpc8t\") pod \"aws-ebs-csi-driver-node-nh7zz\" (UID: \"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.813225 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.813188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-ovn-node-metrics-cert\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.813777 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.813750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6gm\" (UniqueName: \"kubernetes.io/projected/4e57a037-dd6f-4793-bfa6-6145355fa80f-kube-api-access-2n6gm\") pod \"node-ca-sh2lt\" (UID: \"4e57a037-dd6f-4793-bfa6-6145355fa80f\") " pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.817288 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.817249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlstf\" (UniqueName: \"kubernetes.io/projected/37aa38d5-5f42-4c1a-b3ab-82fe91a37aca-kube-api-access-zlstf\") pod \"tuned-82vhr\" (UID: \"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca\") " pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.817690 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.817667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gn6k\" (UniqueName: \"kubernetes.io/projected/ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5-kube-api-access-8gn6k\") pod \"multus-wb5fb\" (UID: \"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5\") " pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.818488 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.818461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl8tl\" (UniqueName: \"kubernetes.io/projected/6abe718b-42b3-4ae8-8051-8bb46abd4db4-kube-api-access-kl8tl\") pod \"multus-additional-cni-plugins-wnjtb\" (UID: \"6abe718b-42b3-4ae8-8051-8bb46abd4db4\") " pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.819471 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.819449 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjn4n\" (UniqueName: \"kubernetes.io/projected/71f8cf88-2dcc-4d71-a25b-b9bd281e00e1-kube-api-access-kjn4n\") pod \"ovnkube-node-z7967\" (UID: \"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.883477 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.883443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:08.894223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.894199 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w85q8" Jul 28 12:03:08.897735 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:08.897690 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d522fb7_96d0_40b8_aa15_33ca968f946c.slice/crio-e578342c65a42b27bfb89f81f41d18477c7dec7993f4a41056ad9f37f02a1bb5 WatchSource:0}: Error finding container e578342c65a42b27bfb89f81f41d18477c7dec7993f4a41056ad9f37f02a1bb5: Status 404 returned error can't find the container with id e578342c65a42b27bfb89f81f41d18477c7dec7993f4a41056ad9f37f02a1bb5 Jul 28 12:03:08.904944 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:08.904917 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea75a61_bc22_494b_8843_902854593d6c.slice/crio-e89b6a75b8fbcec7881a6cd81e0a18e7556324a0bd9f0f8d34d8f2097db2287d WatchSource:0}: Error finding container e89b6a75b8fbcec7881a6cd81e0a18e7556324a0bd9f0f8d34d8f2097db2287d: Status 404 returned error can't find the container with id e89b6a75b8fbcec7881a6cd81e0a18e7556324a0bd9f0f8d34d8f2097db2287d Jul 28 12:03:08.907945 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.907687 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ndsld" Jul 28 12:03:08.915336 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.915313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-82vhr" Jul 28 12:03:08.920255 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:08.920229 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf5bafc_60f8_4a8e_964c_b697c819f0f4.slice/crio-d4a7fec1f02a29a2b451ffe4945703e611fd849b141bf450f4f3cfa984ab65fb WatchSource:0}: Error finding container d4a7fec1f02a29a2b451ffe4945703e611fd849b141bf450f4f3cfa984ab65fb: Status 404 returned error can't find the container with id d4a7fec1f02a29a2b451ffe4945703e611fd849b141bf450f4f3cfa984ab65fb Jul 28 12:03:08.925021 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.924999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sh2lt" Jul 28 12:03:08.933242 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.933136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" Jul 28 12:03:08.934485 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:08.934450 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37aa38d5_5f42_4c1a_b3ab_82fe91a37aca.slice/crio-9934518f2c674d82aeb09dbf2be9b85fbce6ddd0996631c066bf3cc4716d9dc7 WatchSource:0}: Error finding container 9934518f2c674d82aeb09dbf2be9b85fbce6ddd0996631c066bf3cc4716d9dc7: Status 404 returned error can't find the container with id 9934518f2c674d82aeb09dbf2be9b85fbce6ddd0996631c066bf3cc4716d9dc7 Jul 28 12:03:08.943510 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.943041 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wb5fb" Jul 28 12:03:08.946761 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:08.946731 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abe718b_42b3_4ae8_8051_8bb46abd4db4.slice/crio-6e39a74379c4a8828fe0bf114f91d3c248a45d2b88f2088b38418fe26551b9e2 WatchSource:0}: Error finding container 6e39a74379c4a8828fe0bf114f91d3c248a45d2b88f2088b38418fe26551b9e2: Status 404 returned error can't find the container with id 6e39a74379c4a8828fe0bf114f91d3c248a45d2b88f2088b38418fe26551b9e2 Jul 28 12:03:08.950044 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.950024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:08.955459 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:08.955427 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2fb42a_86c5_4534_a6b7_88d8f2a5d1a5.slice/crio-4442d8f018a7039459395dd317480734926959f5ca0c7305a2be2cf29bcff9de WatchSource:0}: Error finding container 4442d8f018a7039459395dd317480734926959f5ca0c7305a2be2cf29bcff9de: Status 404 returned error can't find the container with id 4442d8f018a7039459395dd317480734926959f5ca0c7305a2be2cf29bcff9de Jul 28 12:03:08.956390 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:08.956210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" Jul 28 12:03:08.961635 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:08.961611 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f8cf88_2dcc_4d71_a25b_b9bd281e00e1.slice/crio-93adf230ae226849b53567b5e6508092619da72f0b779d5e59cea3637f1fdd5c WatchSource:0}: Error finding container 93adf230ae226849b53567b5e6508092619da72f0b779d5e59cea3637f1fdd5c: Status 404 returned error can't find the container with id 93adf230ae226849b53567b5e6508092619da72f0b779d5e59cea3637f1fdd5c Jul 28 12:03:09.197615 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.197522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:09.197764 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.197683 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:09.197819 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.197772 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs podName:0afc5b7d-6f03-4e22-b639-88fa243b94d3 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:10.197749907 +0000 UTC m=+4.014679718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs") pod "network-metrics-daemon-gklcw" (UID: "0afc5b7d-6f03-4e22-b639-88fa243b94d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:09.298836 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.298799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:09.299026 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.298942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:09.299115 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.299023 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:03:09.299115 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.299054 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:03:09.299115 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.299070 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bqxhc for pod openshift-network-diagnostics/network-check-target-hmlhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:09.299266 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.299112 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:09.299266 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.299148 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc podName:12686a8e-d528-4138-a8ff-0eaf60244bd2 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:10.299126792 +0000 UTC m=+4.116056609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqxhc" (UniqueName: "kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc") pod "network-check-target-hmlhv" (UID: "12686a8e-d528-4138-a8ff-0eaf60244bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:09.299266 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.299174 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret podName:caf368fa-46b2-4bbc-a2a0-d3f35022719a nodeName:}" failed. No retries permitted until 2026-07-28 12:03:10.299157422 +0000 UTC m=+4.116087226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret") pod "global-pull-secret-syncer-4mk2d" (UID: "caf368fa-46b2-4bbc-a2a0-d3f35022719a") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:09.643698 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.643655 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 11:58:09 +0000 UTC" deadline="2028-03-06 16:07:34.501939593 +0000 UTC" Jul 28 12:03:09.643698 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.643695 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14092h4m24.858249058s" Jul 28 12:03:09.774949 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.774859 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:09.774949 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.774905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:09.775132 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.775036 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:09.775181 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:09.775140 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:09.790033 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.790002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerStarted","Data":"6e39a74379c4a8828fe0bf114f91d3c248a45d2b88f2088b38418fe26551b9e2"} Jul 28 12:03:09.791242 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.791205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sh2lt" event={"ID":"4e57a037-dd6f-4793-bfa6-6145355fa80f","Type":"ContainerStarted","Data":"a19e7baf24d849820b679b8c3ab97a81910afcef5a203923aec3ce368123c511"} Jul 28 12:03:09.792484 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.792456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ndsld" event={"ID":"dcf5bafc-60f8-4a8e-964c-b697c819f0f4","Type":"ContainerStarted","Data":"d4a7fec1f02a29a2b451ffe4945703e611fd849b141bf450f4f3cfa984ab65fb"} Jul 28 12:03:09.793639 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.793607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w85q8" event={"ID":"1ea75a61-bc22-494b-8843-902854593d6c","Type":"ContainerStarted","Data":"e89b6a75b8fbcec7881a6cd81e0a18e7556324a0bd9f0f8d34d8f2097db2287d"} Jul 28 12:03:09.794824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.794800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wb5fb" event={"ID":"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5","Type":"ContainerStarted","Data":"4442d8f018a7039459395dd317480734926959f5ca0c7305a2be2cf29bcff9de"} Jul 28 12:03:09.796095 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.796072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-82vhr" event={"ID":"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca","Type":"ContainerStarted","Data":"9934518f2c674d82aeb09dbf2be9b85fbce6ddd0996631c066bf3cc4716d9dc7"} Jul 28 12:03:09.797439 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.797412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wr2z4" event={"ID":"5d522fb7-96d0-40b8-aa15-33ca968f946c","Type":"ContainerStarted","Data":"e578342c65a42b27bfb89f81f41d18477c7dec7993f4a41056ad9f37f02a1bb5"} Jul 28 12:03:09.798663 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.798638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" event={"ID":"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b","Type":"ContainerStarted","Data":"c7d08499c2197732c39988654bb199ca1ddf73a34ffd49e7aa28ea78aa24adce"} Jul 28 12:03:09.799898 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:09.799852 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"93adf230ae226849b53567b5e6508092619da72f0b779d5e59cea3637f1fdd5c"} Jul 28 12:03:10.205852 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:10.205748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:10.206037 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.205943 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:10.206037 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.206011 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs podName:0afc5b7d-6f03-4e22-b639-88fa243b94d3 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:12.205991329 +0000 UTC m=+6.022921151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs") pod "network-metrics-daemon-gklcw" (UID: "0afc5b7d-6f03-4e22-b639-88fa243b94d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:10.307312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:10.307384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.307631 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.307697 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret podName:caf368fa-46b2-4bbc-a2a0-d3f35022719a nodeName:}" failed. No retries permitted until 2026-07-28 12:03:12.307677725 +0000 UTC m=+6.124607549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret") pod "global-pull-secret-syncer-4mk2d" (UID: "caf368fa-46b2-4bbc-a2a0-d3f35022719a") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.308147 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.308171 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.308184 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bqxhc for pod openshift-network-diagnostics/network-check-target-hmlhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:10.308272 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.308236 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc podName:12686a8e-d528-4138-a8ff-0eaf60244bd2 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:12.308219583 +0000 UTC m=+6.125149398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqxhc" (UniqueName: "kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc") pod "network-check-target-hmlhv" (UID: "12686a8e-d528-4138-a8ff-0eaf60244bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:10.643988 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:10.643860 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 11:58:09 +0000 UTC" deadline="2028-02-10 16:48:56.494603428 +0000 UTC" Jul 28 12:03:10.643988 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:10.643944 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13492h45m45.8506643s" Jul 28 12:03:10.778813 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:10.778559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:10.778983 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:10.778961 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:11.774668 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:11.774584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:11.775209 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:11.774740 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:11.775209 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:11.775133 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:11.775340 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:11.775252 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:11.816398 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:11.816364 2564 generic.go:356] "Generic (PLEG): container finished" podID="c9adaaed0eb9e1e379ca9054dcfb08b6" containerID="ea0a7a604c23e9d0ecf717a804dce77f2c5c60d631ba5866c6388fc2e49c39bd" exitCode=0 Jul 28 12:03:11.816643 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:11.816584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" event={"ID":"c9adaaed0eb9e1e379ca9054dcfb08b6","Type":"ContainerDied","Data":"ea0a7a604c23e9d0ecf717a804dce77f2c5c60d631ba5866c6388fc2e49c39bd"} Jul 28 12:03:11.820435 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:11.820409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" event={"ID":"5b4c93f4e70558b77ddac96ac849afbc","Type":"ContainerStarted","Data":"f2673294e12f6faf504bb403d40ada8898d01623b5777754e24b3476482c4f32"} Jul 28 12:03:12.222764 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:12.222475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:12.222764 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.222638 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:12.222764 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.222705 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs podName:0afc5b7d-6f03-4e22-b639-88fa243b94d3 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:16.222686031 +0000 UTC m=+10.039615828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs") pod "network-metrics-daemon-gklcw" (UID: "0afc5b7d-6f03-4e22-b639-88fa243b94d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:12.323416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:12.323509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.323663 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.323685 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.323699 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bqxhc for pod openshift-network-diagnostics/network-check-target-hmlhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.323761 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc podName:12686a8e-d528-4138-a8ff-0eaf60244bd2 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:16.323742404 +0000 UTC m=+10.140672202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqxhc" (UniqueName: "kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc") pod "network-check-target-hmlhv" (UID: "12686a8e-d528-4138-a8ff-0eaf60244bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.324198 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:12.324298 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.324257 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret podName:caf368fa-46b2-4bbc-a2a0-d3f35022719a nodeName:}" failed. No retries permitted until 2026-07-28 12:03:16.324239444 +0000 UTC m=+10.141169243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret") pod "global-pull-secret-syncer-4mk2d" (UID: "caf368fa-46b2-4bbc-a2a0-d3f35022719a") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:12.777462 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:12.777429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:12.778004 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:12.777551 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:13.776296 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:13.775545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:13.776296 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:13.775685 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:13.776296 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:13.776100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:13.776296 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:13.776228 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:14.776710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:14.776132 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:14.776710 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:14.776276 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:15.775525 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:15.775005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:15.775525 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:15.775019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:15.775525 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:15.775156 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:15.775525 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:15.775259 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:16.257773 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:16.257112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:16.257773 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.257311 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:16.257773 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.257385 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs podName:0afc5b7d-6f03-4e22-b639-88fa243b94d3 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:24.257364771 +0000 UTC m=+18.074294581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs") pod "network-metrics-daemon-gklcw" (UID: "0afc5b7d-6f03-4e22-b639-88fa243b94d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:16.358104 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:16.358060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:16.358316 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:16.358132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:16.358316 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.358254 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:16.358316 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.358275 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:03:16.358316 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.358303 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:03:16.358316 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.358317 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bqxhc for pod openshift-network-diagnostics/network-check-target-hmlhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:16.358581 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.358332 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret podName:caf368fa-46b2-4bbc-a2a0-d3f35022719a nodeName:}" failed. No retries permitted until 2026-07-28 12:03:24.358311271 +0000 UTC m=+18.175241081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret") pod "global-pull-secret-syncer-4mk2d" (UID: "caf368fa-46b2-4bbc-a2a0-d3f35022719a") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:16.358581 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.358373 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc podName:12686a8e-d528-4138-a8ff-0eaf60244bd2 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:24.358354708 +0000 UTC m=+18.175284518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqxhc" (UniqueName: "kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc") pod "network-check-target-hmlhv" (UID: "12686a8e-d528-4138-a8ff-0eaf60244bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:16.776281 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:16.776253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:16.776469 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:16.776373 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:17.775495 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:17.775458 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:17.775963 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:17.775588 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:17.775963 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:17.775956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:17.776076 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:17.776041 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:18.775586 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:18.775126 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:18.775586 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:18.775249 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:19.775409 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:19.775371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:19.775409 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:19.775416 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:19.775636 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:19.775502 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:19.775962 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:19.775659 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:20.775434 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:20.775360 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:20.775584 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:20.775496 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:21.774843 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:21.774813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:21.775307 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:21.774815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:21.775307 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:21.774952 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:21.775307 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:21.774993 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:22.774635 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:22.774594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:22.774818 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:22.774743 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:23.774666 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:23.774631 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:23.775090 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:23.774635 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:23.775090 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:23.774750 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:23.775090 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:23.774851 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:24.323286 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:24.323257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:24.323437 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.323370 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:24.323437 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.323422 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs podName:0afc5b7d-6f03-4e22-b639-88fa243b94d3 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:40.323406885 +0000 UTC m=+34.140336680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs") pod "network-metrics-daemon-gklcw" (UID: "0afc5b7d-6f03-4e22-b639-88fa243b94d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:24.424068 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:24.424033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:24.424231 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:24.424083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:24.424231 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.424212 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:24.424315 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.424230 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:03:24.424315 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.424258 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:03:24.424315 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.424273 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bqxhc for pod openshift-network-diagnostics/network-check-target-hmlhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:24.424315 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.424272 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret podName:caf368fa-46b2-4bbc-a2a0-d3f35022719a nodeName:}" failed. No retries permitted until 2026-07-28 12:03:40.424254476 +0000 UTC m=+34.241184272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret") pod "global-pull-secret-syncer-4mk2d" (UID: "caf368fa-46b2-4bbc-a2a0-d3f35022719a") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:24.424457 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.424335 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc podName:12686a8e-d528-4138-a8ff-0eaf60244bd2 nodeName:}" failed. No retries permitted until 2026-07-28 12:03:40.42431928 +0000 UTC m=+34.241249079 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqxhc" (UniqueName: "kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc") pod "network-check-target-hmlhv" (UID: "12686a8e-d528-4138-a8ff-0eaf60244bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:24.774590 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:24.774558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:24.774764 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:24.774666 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:25.774752 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:25.774716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:25.775182 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:25.774836 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:25.775182 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:25.774924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:25.775182 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:25.775049 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:26.776389 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:26.776050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:26.776389 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:26.776168 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:27.774996 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:27.774960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:27.775175 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:27.775071 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:27.775175 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:27.774973 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:27.775302 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:27.775233 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:28.775657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.775585 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:28.776102 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:28.775718 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:28.880361 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.879616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" event={"ID":"c9adaaed0eb9e1e379ca9054dcfb08b6","Type":"ContainerStarted","Data":"b3bf34feb76ba24451df495ae54b0f05309ddc10c81d84893d59fc433ad2ac57"} Jul 28 12:03:28.884294 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.884189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w85q8" event={"ID":"1ea75a61-bc22-494b-8843-902854593d6c","Type":"ContainerStarted","Data":"d5d4121fd55388e03bec4d0a18b54759c7b5b62fae3880ea91a91af3f71aba81"} Jul 28 12:03:28.886450 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.886365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-82vhr" event={"ID":"37aa38d5-5f42-4c1a-b3ab-82fe91a37aca","Type":"ContainerStarted","Data":"7f6077b8849cebda9636b569a305ecfc928e719f3036f2924a833d48f7007ae1"} Jul 28 12:03:28.888562 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.888535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wr2z4" event={"ID":"5d522fb7-96d0-40b8-aa15-33ca968f946c","Type":"ContainerStarted","Data":"94a9c8d11fe842c6b4db861232db1bd55d9d046a0e38d48c01b033346fbeb74b"} Jul 28 12:03:28.900984 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.900926 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" podStartSLOduration=20.900904659 podStartE2EDuration="20.900904659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:28.897034291 +0000 UTC m=+22.713964122" watchObservedRunningTime="2026-07-28 12:03:28.900904659 +0000 UTC m=+22.717834479" Jul 28 12:03:28.901440 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.901408 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" podStartSLOduration=20.901395793 podStartE2EDuration="20.901395793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:11.856807886 +0000 UTC m=+5.673737709" watchObservedRunningTime="2026-07-28 12:03:28.901395793 +0000 UTC m=+22.718325610" Jul 28 12:03:28.935639 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.935599 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w85q8" podStartSLOduration=2.163205637 podStartE2EDuration="21.935507612s" totalImagesPullingTime="19.772301975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:28.935459071 +0000 UTC m=+22.752388889" watchObservedRunningTime="2026-07-28 12:03:28.935507612 +0000 UTC m=+22.752437429" Jul 28 12:03:28.935948 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.935926 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wr2z4" podStartSLOduration=10.307382314 podStartE2EDuration="21.935919164s" totalImagesPullingTime="11.62853685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:28.91440956 +0000 UTC m=+22.731339378" watchObservedRunningTime="2026-07-28 12:03:28.935919164 +0000 UTC m=+22.752848981" Jul 28 12:03:28.954195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:28.954156 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-82vhr" podStartSLOduration=2.209172707 podStartE2EDuration="21.954141886s" totalImagesPullingTime="19.744969179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:28.953621382 +0000 UTC m=+22.770551201" watchObservedRunningTime="2026-07-28 12:03:28.954141886 +0000 UTC m=+22.771071704" Jul 28 12:03:29.775526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.775336 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:29.775748 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.775391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:29.775748 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:29.775629 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:29.775748 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:29.775668 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:29.893305 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893275 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:03:29.893615 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893596 2564 generic.go:356] "Generic (PLEG): container finished" podID="71f8cf88-2dcc-4d71-a25b-b9bd281e00e1" containerID="66810494778af1a5ad4ab00389c4bac36f1d1d8c9ba65171b8cfb983010a4ea8" exitCode=1 Jul 28 12:03:29.893693 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"21cee14b7607f054ae5ce7e4ce8a64562e5046779736c501811c6e02114daeda"} Jul 28 12:03:29.893766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"1d9ff485ea2ce847f63274b83f82534e65e14a4fc9b58c9739c90d0334ec40f2"} Jul 28 12:03:29.893814 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"d6c9ec5f3c077ed71641d2840ac4996c95e56d02727b7f9ea1ec21334380947f"} Jul 28 12:03:29.893863 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"394ab8c3ef04af2bde6ee09ea16c9b211bf1369e7ee2b54f694af795674a7171"} Jul 28 12:03:29.893922 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"b0c0be9047419fd858c8745252ef322a0ae2ce717e1fe27567a4c14bb5faa4d0"} Jul 28 12:03:29.893956 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.893930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerDied","Data":"66810494778af1a5ad4ab00389c4bac36f1d1d8c9ba65171b8cfb983010a4ea8"} Jul 28 12:03:29.895211 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.895189 2564 generic.go:356] "Generic (PLEG): container finished" podID="6abe718b-42b3-4ae8-8051-8bb46abd4db4" containerID="16f12266aacc84697c6a4a05ee3eca480464af75454da2fa1e7141d6a9cb8efa" exitCode=0 Jul 28 12:03:29.895313 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.895260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerDied","Data":"16f12266aacc84697c6a4a05ee3eca480464af75454da2fa1e7141d6a9cb8efa"} Jul 28 12:03:29.896849 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.896829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sh2lt" event={"ID":"4e57a037-dd6f-4793-bfa6-6145355fa80f","Type":"ContainerStarted","Data":"3d0f568c4968edc96a02e7c223734cd39eb7ff18c3118dea1a43c90f9f1884f7"} Jul 28 12:03:29.898356 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.898335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wb5fb" event={"ID":"ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5","Type":"ContainerStarted","Data":"cc2ad9c10877f57a4af803c1f122f96291422a1878aee2ee01587f8a0e272cfe"} Jul 28 12:03:29.899793 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.899770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" event={"ID":"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b","Type":"ContainerStarted","Data":"d341935f7961351b621ac2c8367a62856e4b72db0ca3e966d608207ec1fe3091"} Jul 28 12:03:29.961323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:29.961274 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sh2lt" podStartSLOduration=3.227659695 podStartE2EDuration="22.961257046s" totalImagesPullingTime="19.733597351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:29.938169975 +0000 UTC m=+23.755099792" watchObservedRunningTime="2026-07-28 12:03:29.961257046 +0000 UTC m=+23.778186840" Jul 28 12:03:30.297426 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.297403 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 12:03:30.643039 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.642945 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:03:30.297420768Z","UUID":"4ae8c142-2da1-441b-8714-98ac53738671","Handler":null,"Name":"","Endpoint":""} Jul 28 12:03:30.644617 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.644594 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 12:03:30.644740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.644639 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 12:03:30.775302 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.775272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:30.775475 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:30.775422 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:30.904585 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.904488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ndsld" event={"ID":"dcf5bafc-60f8-4a8e-964c-b697c819f0f4","Type":"ContainerStarted","Data":"643657fd7b63835bb4d63b6197461a9ffb31381f0e579e40077f4fe28c0a0f2f"} Jul 28 12:03:30.907528 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.907489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" event={"ID":"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b","Type":"ContainerStarted","Data":"6f9c903fb7aa610739055fc1873053f2f16f4b088346ccfeff1a2a2953ec916a"} Jul 28 12:03:30.920562 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.920506 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wb5fb" podStartSLOduration=4.161510504 podStartE2EDuration="23.920486014s" totalImagesPullingTime="19.75897551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:29.961416017 +0000 UTC m=+23.778345833" watchObservedRunningTime="2026-07-28 12:03:30.920486014 +0000 UTC m=+24.737415837" Jul 28 12:03:30.920761 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:30.920731 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ndsld" podStartSLOduration=4.163744751 podStartE2EDuration="23.920721322s" totalImagesPullingTime="19.756976571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:30.9204341 +0000 UTC m=+24.737363917" watchObservedRunningTime="2026-07-28 12:03:30.920721322 +0000 UTC m=+24.737651142" Jul 28 12:03:31.774723 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:31.774693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:31.774931 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:31.774695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:31.774931 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:31.774814 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:31.774931 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:31.774869 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:31.911943 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:31.911899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" event={"ID":"6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b","Type":"ContainerStarted","Data":"b3c837909a8e8e2b8ebb2146ef7908d2812711388e0f6c40768194594cf18fff"} Jul 28 12:03:31.915270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:31.915243 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:03:31.915708 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:31.915686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"ee42ea8265fc547bfca20a04f714053291111d1d445e8e67c6e37915ada8af6b"} Jul 28 12:03:31.950661 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:31.950605 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nh7zz" podStartSLOduration=3.141405983 podStartE2EDuration="24.95058768s" totalImagesPullingTime="21.809181697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:31.950106567 +0000 UTC m=+25.767036381" watchObservedRunningTime="2026-07-28 12:03:31.95058768 +0000 UTC m=+25.767517498" Jul 28 12:03:32.775038 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:32.774787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:32.775229 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:32.775145 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:33.088860 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.088774 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w85q8_1ea75a61-bc22-494b-8843-902854593d6c/dns-node-resolver/0.log" Jul 28 12:03:33.772290 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.772263 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:33.772894 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.772851 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:33.775039 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.775019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:33.775153 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.775041 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:33.775153 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:33.775100 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:33.775262 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:33.775223 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:33.867201 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.867181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sh2lt_4e57a037-dd6f-4793-bfa6-6145355fa80f/node-ca/0.log" Jul 28 12:03:33.921397 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.921275 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:33.921931 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:33.921912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wr2z4" Jul 28 12:03:34.775294 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.775109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:34.776114 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:34.775387 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:34.926133 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.926102 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:03:34.926454 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.926431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"63238a4e386b5d193f1deb09ee7784fb7141f6774762cbc6a1bbd38ce323e526"} Jul 28 12:03:34.926720 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.926698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:34.926836 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.926773 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:34.926948 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.926931 2564 scope.go:117] "RemoveContainer" containerID="66810494778af1a5ad4ab00389c4bac36f1d1d8c9ba65171b8cfb983010a4ea8" Jul 28 12:03:34.928699 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.928674 2564 generic.go:356] "Generic (PLEG): container finished" podID="6abe718b-42b3-4ae8-8051-8bb46abd4db4" containerID="794d55459c1d1fdebb5f69d8efc1998c5a04d0a419f433458d41147f9edde6d4" exitCode=0 Jul 28 12:03:34.928795 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.928762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerDied","Data":"794d55459c1d1fdebb5f69d8efc1998c5a04d0a419f433458d41147f9edde6d4"} Jul 28 12:03:34.944664 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:34.944648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:35.775299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.775108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:35.775639 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.775116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:35.775639 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:35.775377 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:35.775639 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:35.775479 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:35.887210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.887131 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hmlhv"] Jul 28 12:03:35.887738 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.887718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gklcw"] Jul 28 12:03:35.889867 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.889842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4mk2d"] Jul 28 12:03:35.890003 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.889954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:35.890081 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:35.890060 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:35.934839 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.934814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:03:35.935258 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.935231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" event={"ID":"71f8cf88-2dcc-4d71-a25b-b9bd281e00e1","Type":"ContainerStarted","Data":"97c66998a8cbfe4fe413d80b750df4c9ff4f2bfc105cc1f20fbd4bdea4d86050"} Jul 28 12:03:35.935509 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.935486 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:35.937448 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.937430 2564 generic.go:356] "Generic (PLEG): container finished" podID="6abe718b-42b3-4ae8-8051-8bb46abd4db4" containerID="4488824a6145b21f0ce089d4f0206a7f63bf17aeac339e64bd974d7ae4c86687" exitCode=0 Jul 28 12:03:35.937556 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.937470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerDied","Data":"4488824a6145b21f0ce089d4f0206a7f63bf17aeac339e64bd974d7ae4c86687"} Jul 28 12:03:35.937556 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.937502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:35.937556 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.937513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:35.937841 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:35.937821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:35.937963 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:35.937946 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:35.951579 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.951558 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:03:35.976018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:35.975969 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" podStartSLOduration=9.190560516 podStartE2EDuration="28.975957319s" totalImagesPullingTime="19.785396803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:35.975560395 +0000 UTC m=+29.792490222" watchObservedRunningTime="2026-07-28 12:03:35.975957319 +0000 UTC m=+29.792887135" Jul 28 12:03:36.941596 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:36.941519 2564 generic.go:356] "Generic (PLEG): container finished" podID="6abe718b-42b3-4ae8-8051-8bb46abd4db4" containerID="9bc422edcf4cfd26dc4392f1711eb1754e039a576ffc8647c25727b0cefd3291" exitCode=0 Jul 28 12:03:36.942013 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:36.941599 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerDied","Data":"9bc422edcf4cfd26dc4392f1711eb1754e039a576ffc8647c25727b0cefd3291"} Jul 28 12:03:37.774832 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:37.774802 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:37.774998 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:37.774802 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:37.774998 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:37.774938 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:37.775102 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:37.774811 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:37.775102 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:37.775028 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:37.775102 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:37.775083 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:39.774910 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:39.774850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:39.774910 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:39.774912 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:39.775574 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:39.774989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:39.775574 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:39.774996 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:39.775574 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:39.775056 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:39.775574 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:39.775144 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:40.340543 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:40.340503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:40.340758 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.340667 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:40.340821 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.340748 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs podName:0afc5b7d-6f03-4e22-b639-88fa243b94d3 nodeName:}" failed. No retries permitted until 2026-07-28 12:04:12.34072649 +0000 UTC m=+66.157656302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs") pod "network-metrics-daemon-gklcw" (UID: "0afc5b7d-6f03-4e22-b639-88fa243b94d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 12:03:40.440986 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:40.440951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:40.441146 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:40.441016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:40.441146 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.441110 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:40.441146 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.441127 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 12:03:40.441146 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.441143 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 12:03:40.441308 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.441157 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bqxhc for pod openshift-network-diagnostics/network-check-target-hmlhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:40.441308 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.441176 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret podName:caf368fa-46b2-4bbc-a2a0-d3f35022719a nodeName:}" failed. No retries permitted until 2026-07-28 12:04:12.441157293 +0000 UTC m=+66.258087107 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret") pod "global-pull-secret-syncer-4mk2d" (UID: "caf368fa-46b2-4bbc-a2a0-d3f35022719a") : object "kube-system"/"original-pull-secret" not registered Jul 28 12:03:40.441308 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:40.441192 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc podName:12686a8e-d528-4138-a8ff-0eaf60244bd2 nodeName:}" failed. No retries permitted until 2026-07-28 12:04:12.441182776 +0000 UTC m=+66.258112574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bqxhc" (UniqueName: "kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc") pod "network-check-target-hmlhv" (UID: "12686a8e-d528-4138-a8ff-0eaf60244bd2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 12:03:41.774817 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:41.774782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:41.774817 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:41.774809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:41.775366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:41.774917 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:41.775366 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:41.774945 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:41.775366 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:41.775044 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:41.775366 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:41.775111 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:41.957346 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:41.957309 2564 generic.go:356] "Generic (PLEG): container finished" podID="6abe718b-42b3-4ae8-8051-8bb46abd4db4" containerID="44d9e1f5023838bf6f860d782bee2e9b2a557d5a8d2192d7931e1f4c2b7acf41" exitCode=0 Jul 28 12:03:41.957506 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:41.957363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerDied","Data":"44d9e1f5023838bf6f860d782bee2e9b2a557d5a8d2192d7931e1f4c2b7acf41"} Jul 28 12:03:42.962470 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:42.962437 2564 generic.go:356] "Generic (PLEG): container finished" podID="6abe718b-42b3-4ae8-8051-8bb46abd4db4" containerID="c4f4a0f1158e5cd28ab846864f7b81db57d17d0ace194d18d9ca87fb82e0d6a0" exitCode=0 Jul 28 12:03:42.962839 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:42.962484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerDied","Data":"c4f4a0f1158e5cd28ab846864f7b81db57d17d0ace194d18d9ca87fb82e0d6a0"} Jul 28 12:03:43.774894 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:43.774690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:43.775070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:43.774710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:43.775070 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:43.775006 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:43.775070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:43.774748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:43.775070 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:43.775060 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:43.775198 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:43.775150 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:43.967560 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:43.967505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" event={"ID":"6abe718b-42b3-4ae8-8051-8bb46abd4db4","Type":"ContainerStarted","Data":"97705ad06e7415e1260a9cd7121454b640901dea4ac90e34776e3142b6238dd0"} Jul 28 12:03:43.988602 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:43.988559 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wnjtb" podStartSLOduration=7.645112554 podStartE2EDuration="36.988543455s" totalImagesPullingTime="29.343430901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:43.988281159 +0000 UTC m=+37.805210970" watchObservedRunningTime="2026-07-28 12:03:43.988543455 +0000 UTC m=+37.805473272" Jul 28 12:03:45.774769 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:45.774737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:45.775262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:45.774833 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:45.775262 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:45.774835 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:45.775262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:45.774837 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:45.775262 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:45.774964 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:45.775262 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:45.775022 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:47.775236 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:47.775195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:47.775236 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:47.775219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:47.775682 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:47.775307 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:47.775682 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:47.775325 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:47.775682 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:47.775376 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:47.775682 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:47.775451 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:49.775109 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:49.775074 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:49.775564 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:49.775074 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:49.775564 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:49.775196 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:49.775564 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:49.775080 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:49.775564 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:49.775262 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:49.775564 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:49.775332 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:51.774950 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:51.774917 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:51.774950 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:51.774940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:51.775438 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:51.775030 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:51.775438 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:51.775101 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:51.775438 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:51.775119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:51.775438 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:51.775196 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:53.774550 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:53.774522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:53.774968 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:53.774613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:53.774968 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:53.774615 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:53.774968 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:53.774652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:53.774968 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:53.774727 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:53.774968 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:53.774774 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:55.775173 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:55.775136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:55.775522 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:55.775230 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hmlhv" podUID="12686a8e-d528-4138-a8ff-0eaf60244bd2" Jul 28 12:03:55.775522 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:55.775144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:55.775522 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:55.775251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:55.775522 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:55.775289 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mk2d" podUID="caf368fa-46b2-4bbc-a2a0-d3f35022719a" Jul 28 12:03:55.775522 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:03:55.775342 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gklcw" podUID="0afc5b7d-6f03-4e22-b639-88fa243b94d3" Jul 28 12:03:57.525631 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.525603 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeReady" Jul 28 12:03:57.526049 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.525743 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 12:03:57.586749 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.586721 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kr5xt"] Jul 28 12:03:57.608317 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.608240 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vqt77"] Jul 28 12:03:57.608490 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.608477 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.611431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.611393 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 12:03:57.611582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.611500 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qp4ft\"" Jul 28 12:03:57.611582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.611500 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 12:03:57.630251 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.630229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqt77"] Jul 28 12:03:57.630375 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.630260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kr5xt"] Jul 28 12:03:57.630375 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.630354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:57.633099 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.633069 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 12:03:57.633263 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.633246 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 12:03:57.633480 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.633469 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g7blm\"" Jul 28 12:03:57.633795 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.633783 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 12:03:57.665658 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.665635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3297c9b-23e3-49fc-8b8e-18d121542221-metrics-tls\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.665766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.665660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3297c9b-23e3-49fc-8b8e-18d121542221-tmp-dir\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.665766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.665677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3297c9b-23e3-49fc-8b8e-18d121542221-config-volume\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.665766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.665691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jw6\" (UniqueName: \"kubernetes.io/projected/d3297c9b-23e3-49fc-8b8e-18d121542221-kube-api-access-89jw6\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.665766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.665707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdnf\" (UniqueName: \"kubernetes.io/projected/bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42-kube-api-access-mqdnf\") pod \"ingress-canary-vqt77\" (UID: \"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42\") " pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:57.665927 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.665826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42-cert\") pod \"ingress-canary-vqt77\" (UID: \"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42\") " pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:57.698518 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.698500 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-trjwp"] Jul 28 12:03:57.719598 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.719571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.722143 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.722121 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-trjwp"] Jul 28 12:03:57.726495 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.726457 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 12:03:57.726590 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.726457 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 12:03:57.726900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.726867 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7cqps\"" Jul 28 12:03:57.727001 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.726973 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 12:03:57.727399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.727381 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 12:03:57.766854 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.766834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aec5250b-4ff2-414b-aab2-a8c726a09765-crio-socket\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.766960 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.766892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42-cert\") pod \"ingress-canary-vqt77\" (UID: \"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42\") " pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:57.766960 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.766910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aec5250b-4ff2-414b-aab2-a8c726a09765-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.766960 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.766946 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx525\" (UniqueName: \"kubernetes.io/projected/aec5250b-4ff2-414b-aab2-a8c726a09765-kube-api-access-jx525\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.767079 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.766984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3297c9b-23e3-49fc-8b8e-18d121542221-metrics-tls\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.767079 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3297c9b-23e3-49fc-8b8e-18d121542221-tmp-dir\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.767079 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3297c9b-23e3-49fc-8b8e-18d121542221-config-volume\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.767200 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jw6\" (UniqueName: \"kubernetes.io/projected/d3297c9b-23e3-49fc-8b8e-18d121542221-kube-api-access-89jw6\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.767242 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aec5250b-4ff2-414b-aab2-a8c726a09765-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.767242 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aec5250b-4ff2-414b-aab2-a8c726a09765-data-volume\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.767336 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdnf\" (UniqueName: \"kubernetes.io/projected/bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42-kube-api-access-mqdnf\") pod \"ingress-canary-vqt77\" (UID: \"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42\") " pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:57.767388 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3297c9b-23e3-49fc-8b8e-18d121542221-tmp-dir\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.767530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.767513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3297c9b-23e3-49fc-8b8e-18d121542221-config-volume\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.770916 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.770896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3297c9b-23e3-49fc-8b8e-18d121542221-metrics-tls\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.770997 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.770940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42-cert\") pod \"ingress-canary-vqt77\" (UID: \"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42\") " pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:57.774917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.774897 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:03:57.774917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.774911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:03:57.775064 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.774911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:03:57.777139 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.777120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdnf\" (UniqueName: \"kubernetes.io/projected/bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42-kube-api-access-mqdnf\") pod \"ingress-canary-vqt77\" (UID: \"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42\") " pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:57.778164 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.778146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jw6\" (UniqueName: \"kubernetes.io/projected/d3297c9b-23e3-49fc-8b8e-18d121542221-kube-api-access-89jw6\") pod \"dns-default-kr5xt\" (UID: \"d3297c9b-23e3-49fc-8b8e-18d121542221\") " pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.778543 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.778522 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 12:03:57.778622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.778564 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 12:03:57.778622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.778577 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 12:03:57.778622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.778534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rnkk\"" Jul 28 12:03:57.778760 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.778619 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfpmm\"" Jul 28 12:03:57.778760 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.778681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 12:03:57.868044 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.867948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aec5250b-4ff2-414b-aab2-a8c726a09765-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.868044 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.867984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aec5250b-4ff2-414b-aab2-a8c726a09765-data-volume\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.868270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.868133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aec5250b-4ff2-414b-aab2-a8c726a09765-crio-socket\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.868270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.868174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aec5250b-4ff2-414b-aab2-a8c726a09765-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.868270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.868208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx525\" (UniqueName: \"kubernetes.io/projected/aec5250b-4ff2-414b-aab2-a8c726a09765-kube-api-access-jx525\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.868270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.868264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aec5250b-4ff2-414b-aab2-a8c726a09765-data-volume\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.868471 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.868351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aec5250b-4ff2-414b-aab2-a8c726a09765-crio-socket\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.868652 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.868630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aec5250b-4ff2-414b-aab2-a8c726a09765-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.870685 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.870665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aec5250b-4ff2-414b-aab2-a8c726a09765-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.876291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.876273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx525\" (UniqueName: \"kubernetes.io/projected/aec5250b-4ff2-414b-aab2-a8c726a09765-kube-api-access-jx525\") pod \"insights-runtime-extractor-trjwp\" (UID: \"aec5250b-4ff2-414b-aab2-a8c726a09765\") " pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:57.919300 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.919270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kr5xt" Jul 28 12:03:57.938366 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:57.938346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqt77" Jul 28 12:03:58.029649 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:58.029613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-trjwp" Jul 28 12:03:58.151331 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:58.151290 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqt77"] Jul 28 12:03:58.160333 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:58.160308 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae8697e_e6b5_43c7_93e5_e5d8c5ae2c42.slice/crio-798d7eef332b6baf3456b46537beab2786426f92605277d27e5f73bf5f1fd4c3 WatchSource:0}: Error finding container 798d7eef332b6baf3456b46537beab2786426f92605277d27e5f73bf5f1fd4c3: Status 404 returned error can't find the container with id 798d7eef332b6baf3456b46537beab2786426f92605277d27e5f73bf5f1fd4c3 Jul 28 12:03:58.178363 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:58.178331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kr5xt"] Jul 28 12:03:58.181687 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:58.181666 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3297c9b_23e3_49fc_8b8e_18d121542221.slice/crio-89915dfc3ff3cbf93b92e2795fad4b85e8ed61ec0221306e328d18b2cedc01d8 WatchSource:0}: Error finding container 89915dfc3ff3cbf93b92e2795fad4b85e8ed61ec0221306e328d18b2cedc01d8: Status 404 returned error can't find the container with id 89915dfc3ff3cbf93b92e2795fad4b85e8ed61ec0221306e328d18b2cedc01d8 Jul 28 12:03:58.213333 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:58.213301 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-trjwp"] Jul 28 12:03:58.216970 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:03:58.216940 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec5250b_4ff2_414b_aab2_a8c726a09765.slice/crio-17676b8bf0a8800cfe833081ec80e9c1f7306eff7b837f1d9df0a449df901f46 WatchSource:0}: Error finding container 17676b8bf0a8800cfe833081ec80e9c1f7306eff7b837f1d9df0a449df901f46: Status 404 returned error can't find the container with id 17676b8bf0a8800cfe833081ec80e9c1f7306eff7b837f1d9df0a449df901f46 Jul 28 12:03:59.002781 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:59.002734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trjwp" event={"ID":"aec5250b-4ff2-414b-aab2-a8c726a09765","Type":"ContainerStarted","Data":"17676b8bf0a8800cfe833081ec80e9c1f7306eff7b837f1d9df0a449df901f46"} Jul 28 12:03:59.005462 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:59.005382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqt77" event={"ID":"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42","Type":"ContainerStarted","Data":"798d7eef332b6baf3456b46537beab2786426f92605277d27e5f73bf5f1fd4c3"} Jul 28 12:03:59.008465 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:03:59.008428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kr5xt" event={"ID":"d3297c9b-23e3-49fc-8b8e-18d121542221","Type":"ContainerStarted","Data":"89915dfc3ff3cbf93b92e2795fad4b85e8ed61ec0221306e328d18b2cedc01d8"} Jul 28 12:04:00.317284 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.317238 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-df698df75-7sp9r"] Jul 28 12:04:00.319713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.319686 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.326604 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.326313 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 12:04:00.326604 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.326396 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 12:04:00.326604 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.326415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 12:04:00.326604 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.326547 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 12:04:00.328284 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.328265 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 12:04:00.328408 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.328297 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 12:04:00.329148 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.329128 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 12:04:00.329340 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.329322 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vgzjb\"" Jul 28 12:04:00.344911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.344872 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df698df75-7sp9r"] Jul 28 12:04:00.390001 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.389976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-serving-cert\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.390137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.390010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-config\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.390137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.390045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2tw\" (UniqueName: \"kubernetes.io/projected/9abbcf32-0d25-4dca-88df-1fab1334fdcd-kube-api-access-rs2tw\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.390137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.390076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-oauth-serving-cert\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.390137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.390106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-oauth-config\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.390343 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.390175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-service-ca\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.491534 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.491497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-serving-cert\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.491702 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.491540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-config\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.491702 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.491570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2tw\" (UniqueName: \"kubernetes.io/projected/9abbcf32-0d25-4dca-88df-1fab1334fdcd-kube-api-access-rs2tw\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.492622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.491958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-oauth-serving-cert\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.492622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.492252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-oauth-config\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.492622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.492328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-service-ca\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.492622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.492262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-config\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.492914 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.492663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-oauth-serving-cert\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.492976 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.492910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-service-ca\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.494905 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.494859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-serving-cert\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.498960 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.498936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-oauth-config\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.503577 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.503545 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2tw\" (UniqueName: \"kubernetes.io/projected/9abbcf32-0d25-4dca-88df-1fab1334fdcd-kube-api-access-rs2tw\") pod \"console-df698df75-7sp9r\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:00.634040 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:00.633968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:03.572023 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:03.571941 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b"] Jul 28 12:04:03.576378 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:03.576355 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:03.603468 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:03.603432 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 12:04:03.604581 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:03.604554 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bt77x\"" Jul 28 12:04:03.605175 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:03.605155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b"] Jul 28 12:04:03.619697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:03.619657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/606a9748-163b-4e92-bbcd-9c3ab4d3ce91-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8hn4b\" (UID: \"606a9748-163b-4e92-bbcd-9c3ab4d3ce91\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:03.720442 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:03.720407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/606a9748-163b-4e92-bbcd-9c3ab4d3ce91-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8hn4b\" (UID: \"606a9748-163b-4e92-bbcd-9c3ab4d3ce91\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:03.720614 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:03.720540 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 28 12:04:03.720614 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:03.720607 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606a9748-163b-4e92-bbcd-9c3ab4d3ce91-tls-certificates podName:606a9748-163b-4e92-bbcd-9c3ab4d3ce91 nodeName:}" failed. No retries permitted until 2026-07-28 12:04:04.22058676 +0000 UTC m=+58.037516557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/606a9748-163b-4e92-bbcd-9c3ab4d3ce91-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" (UID: "606a9748-163b-4e92-bbcd-9c3ab4d3ce91") : secret "prometheus-operator-admission-webhook-tls" not found Jul 28 12:04:04.117496 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:04.117051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df698df75-7sp9r"] Jul 28 12:04:04.224345 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:04.224261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/606a9748-163b-4e92-bbcd-9c3ab4d3ce91-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8hn4b\" (UID: \"606a9748-163b-4e92-bbcd-9c3ab4d3ce91\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:04.226714 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:04.226685 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/606a9748-163b-4e92-bbcd-9c3ab4d3ce91-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8hn4b\" (UID: \"606a9748-163b-4e92-bbcd-9c3ab4d3ce91\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:04.486126 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:04.486039 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:04.638764 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:04.637365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b"] Jul 28 12:04:04.642569 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:04.642539 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606a9748_163b_4e92_bbcd_9c3ab4d3ce91.slice/crio-c9af278d9fbce5d36a4b3d48cafe19bade496746ac9e991ba37bd17de6529c1f WatchSource:0}: Error finding container c9af278d9fbce5d36a4b3d48cafe19bade496746ac9e991ba37bd17de6529c1f: Status 404 returned error can't find the container with id c9af278d9fbce5d36a4b3d48cafe19bade496746ac9e991ba37bd17de6529c1f Jul 28 12:04:05.030719 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.030678 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trjwp" event={"ID":"aec5250b-4ff2-414b-aab2-a8c726a09765","Type":"ContainerStarted","Data":"b45ac4c18a57aa4ed2019fec8b32b101c73e5b3c804aa0705642a26efc7cb702"} Jul 28 12:04:05.032491 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.032461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" event={"ID":"606a9748-163b-4e92-bbcd-9c3ab4d3ce91","Type":"ContainerStarted","Data":"c9af278d9fbce5d36a4b3d48cafe19bade496746ac9e991ba37bd17de6529c1f"} Jul 28 12:04:05.034900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.034766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqt77" event={"ID":"bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42","Type":"ContainerStarted","Data":"85d94600e4f928271c2aaa4abc300f46bab3a164854e0c92748361e6aa44f9d3"} Jul 28 12:04:05.036742 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.036719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df698df75-7sp9r" event={"ID":"9abbcf32-0d25-4dca-88df-1fab1334fdcd","Type":"ContainerStarted","Data":"f64d4bab7c2b62655fb9040bcb43e2b52886e2fbae1c11859e47c11e105f7f72"} Jul 28 12:04:05.039997 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.039976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kr5xt" event={"ID":"d3297c9b-23e3-49fc-8b8e-18d121542221","Type":"ContainerStarted","Data":"ca152e580ff200a88ac328a5b829fb64ca75ad3de3df42154b8c4db0e1515361"} Jul 28 12:04:05.040106 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.040051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kr5xt" event={"ID":"d3297c9b-23e3-49fc-8b8e-18d121542221","Type":"ContainerStarted","Data":"e5a315d2b587f583032c17aa77037e5dd7b6790ef9271ba43fe376c9263d7b0c"} Jul 28 12:04:05.040706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.040687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kr5xt" Jul 28 12:04:05.089777 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.089715 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kr5xt" podStartSLOduration=2.629853872 podStartE2EDuration="8.089695598s" totalImagesPullingTime="5.459841726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:05.089131313 +0000 UTC m=+58.906061131" watchObservedRunningTime="2026-07-28 12:04:05.089695598 +0000 UTC m=+58.906625415" Jul 28 12:04:05.089942 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:05.089841 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vqt77" podStartSLOduration=2.285177976 podStartE2EDuration="8.089831047s" totalImagesPullingTime="5.804653071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:05.061254665 +0000 UTC m=+58.878184465" watchObservedRunningTime="2026-07-28 12:04:05.089831047 +0000 UTC m=+58.906760865" Jul 28 12:04:07.957064 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:07.957030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7967" Jul 28 12:04:08.687463 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.687428 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ff44d47d7-9vsnq"] Jul 28 12:04:08.691956 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.691929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.703055 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.703022 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 12:04:08.707382 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.707352 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff44d47d7-9vsnq"] Jul 28 12:04:08.765518 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.765486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-config\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.765679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.765533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-oauth-serving-cert\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.765679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.765565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-serving-cert\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.765679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.765605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-oauth-config\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.765679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.765623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-trusted-ca-bundle\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.765679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.765660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-service-ca\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.765922 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.765717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbc6m\" (UniqueName: \"kubernetes.io/projected/84d5cb74-a790-4feb-bcd1-19df7a193ad0-kube-api-access-fbc6m\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.866412 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.866372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbc6m\" (UniqueName: \"kubernetes.io/projected/84d5cb74-a790-4feb-bcd1-19df7a193ad0-kube-api-access-fbc6m\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.866412 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.866419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-config\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.866649 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.866451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-oauth-serving-cert\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.866649 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.866477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-serving-cert\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.866649 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.866526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-oauth-config\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.866649 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.866550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-trusted-ca-bundle\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.866649 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.866574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-service-ca\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.867187 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.867158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-oauth-serving-cert\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.867318 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.867208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-config\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.867318 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.867246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-service-ca\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.867834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.867812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-trusted-ca-bundle\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.869470 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.869447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-oauth-config\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.870063 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.870041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-serving-cert\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:08.879128 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:08.879110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbc6m\" (UniqueName: \"kubernetes.io/projected/84d5cb74-a790-4feb-bcd1-19df7a193ad0-kube-api-access-fbc6m\") pod \"console-6ff44d47d7-9vsnq\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:09.003387 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:09.003343 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:09.240933 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:09.240901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff44d47d7-9vsnq"] Jul 28 12:04:09.244682 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:09.244622 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d5cb74_a790_4feb_bcd1_19df7a193ad0.slice/crio-aead7d94180b344493a5247f3c483ded8d3db898de0ea5aa0d48ce8894ac5fef WatchSource:0}: Error finding container aead7d94180b344493a5247f3c483ded8d3db898de0ea5aa0d48ce8894ac5fef: Status 404 returned error can't find the container with id aead7d94180b344493a5247f3c483ded8d3db898de0ea5aa0d48ce8894ac5fef Jul 28 12:04:10.056766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.056724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff44d47d7-9vsnq" event={"ID":"84d5cb74-a790-4feb-bcd1-19df7a193ad0","Type":"ContainerStarted","Data":"bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712"} Jul 28 12:04:10.057205 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.056789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff44d47d7-9vsnq" event={"ID":"84d5cb74-a790-4feb-bcd1-19df7a193ad0","Type":"ContainerStarted","Data":"aead7d94180b344493a5247f3c483ded8d3db898de0ea5aa0d48ce8894ac5fef"} Jul 28 12:04:10.059050 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.059026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trjwp" event={"ID":"aec5250b-4ff2-414b-aab2-a8c726a09765","Type":"ContainerStarted","Data":"c372644aec4964c1138c1051c2d46efb017be29597d4260bbb27d6a1bdd9809e"} Jul 28 12:04:10.060719 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.060696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" event={"ID":"606a9748-163b-4e92-bbcd-9c3ab4d3ce91","Type":"ContainerStarted","Data":"a1d333cc1b28a66343e1afa0bf93ce906b12ce140dfe1818cec5c60f04e9ca37"} Jul 28 12:04:10.060915 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.060901 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:10.062797 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.062775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df698df75-7sp9r" event={"ID":"9abbcf32-0d25-4dca-88df-1fab1334fdcd","Type":"ContainerStarted","Data":"8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc"} Jul 28 12:04:10.065983 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.065962 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" Jul 28 12:04:10.089099 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.089046 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ff44d47d7-9vsnq" podStartSLOduration=2.0890266730000002 podStartE2EDuration="2.089026673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:10.087892972 +0000 UTC m=+63.904822787" watchObservedRunningTime="2026-07-28 12:04:10.089026673 +0000 UTC m=+63.905956491" Jul 28 12:04:10.161057 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.160996 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8hn4b" podStartSLOduration=2.722951142 podStartE2EDuration="7.160974554s" totalImagesPullingTime="4.438023412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:10.120481963 +0000 UTC m=+63.937411782" watchObservedRunningTime="2026-07-28 12:04:10.160974554 +0000 UTC m=+63.977904372" Jul 28 12:04:10.635115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.635084 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:10.635296 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.635230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:10.640672 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.640650 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:10.708112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.708048 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-df698df75-7sp9r" podStartSLOduration=5.727505781 podStartE2EDuration="10.708026731s" totalImagesPullingTime="4.98052095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:10.160370182 +0000 UTC m=+63.977300013" watchObservedRunningTime="2026-07-28 12:04:10.708026731 +0000 UTC m=+64.524956529" Jul 28 12:04:10.762963 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.762927 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-f2vhq"] Jul 28 12:04:10.765202 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.765180 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.768232 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.768204 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 12:04:10.768843 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.768803 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 12:04:10.769278 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.769260 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 12:04:10.769278 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.769277 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 12:04:10.769443 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.769412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9dmtx\"" Jul 28 12:04:10.769443 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.769436 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 12:04:10.794697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.794672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-f2vhq"] Jul 28 12:04:10.884239 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.884212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd4840bc-966d-4609-b62f-2246082309ed-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.884239 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.884242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5pq\" (UniqueName: \"kubernetes.io/projected/cd4840bc-966d-4609-b62f-2246082309ed-kube-api-access-cv5pq\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.884459 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.884280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd4840bc-966d-4609-b62f-2246082309ed-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.884459 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.884366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd4840bc-966d-4609-b62f-2246082309ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.985678 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.985637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd4840bc-966d-4609-b62f-2246082309ed-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.985838 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.985686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5pq\" (UniqueName: \"kubernetes.io/projected/cd4840bc-966d-4609-b62f-2246082309ed-kube-api-access-cv5pq\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.985838 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.985739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd4840bc-966d-4609-b62f-2246082309ed-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.985838 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.985775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd4840bc-966d-4609-b62f-2246082309ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.986472 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.986447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd4840bc-966d-4609-b62f-2246082309ed-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.988247 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.988226 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd4840bc-966d-4609-b62f-2246082309ed-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.988669 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.988651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd4840bc-966d-4609-b62f-2246082309ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:10.997857 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:10.997829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5pq\" (UniqueName: \"kubernetes.io/projected/cd4840bc-966d-4609-b62f-2246082309ed-kube-api-access-cv5pq\") pod \"prometheus-operator-84d69f794d-f2vhq\" (UID: \"cd4840bc-966d-4609-b62f-2246082309ed\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:11.070031 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:11.070001 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:11.076990 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:11.076966 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" Jul 28 12:04:12.400930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.400894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:04:12.404568 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.404539 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 12:04:12.413651 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.413621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afc5b7d-6f03-4e22-b639-88fa243b94d3-metrics-certs\") pod \"network-metrics-daemon-gklcw\" (UID: \"0afc5b7d-6f03-4e22-b639-88fa243b94d3\") " pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:04:12.501535 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.501498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:04:12.501729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.501609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:04:12.501807 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.501514 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfpmm\"" Jul 28 12:04:12.506252 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.506217 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 12:04:12.506437 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.506217 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 12:04:12.508390 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.508356 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gklcw" Jul 28 12:04:12.515639 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.515613 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caf368fa-46b2-4bbc-a2a0-d3f35022719a-original-pull-secret\") pod \"global-pull-secret-syncer-4mk2d\" (UID: \"caf368fa-46b2-4bbc-a2a0-d3f35022719a\") " pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:04:12.515789 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.515752 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 12:04:12.525920 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.525867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqxhc\" (UniqueName: \"kubernetes.io/projected/12686a8e-d528-4138-a8ff-0eaf60244bd2-kube-api-access-bqxhc\") pod \"network-check-target-hmlhv\" (UID: \"12686a8e-d528-4138-a8ff-0eaf60244bd2\") " pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:04:12.792567 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.792535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mk2d" Jul 28 12:04:12.807106 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.807077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rnkk\"" Jul 28 12:04:12.813771 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.813743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:04:12.978020 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:12.977994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gklcw"] Jul 28 12:04:13.007646 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:13.007600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-f2vhq"] Jul 28 12:04:13.029972 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:13.029943 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4mk2d"] Jul 28 12:04:13.038578 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:13.038558 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hmlhv"] Jul 28 12:04:13.163082 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:13.163051 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afc5b7d_6f03_4e22_b639_88fa243b94d3.slice/crio-81822aab92c3ad6ae4d0f4e22710a458d15838f877909841f978cdb5e2707b84 WatchSource:0}: Error finding container 81822aab92c3ad6ae4d0f4e22710a458d15838f877909841f978cdb5e2707b84: Status 404 returned error can't find the container with id 81822aab92c3ad6ae4d0f4e22710a458d15838f877909841f978cdb5e2707b84 Jul 28 12:04:13.163652 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:13.163609 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4840bc_966d_4609_b62f_2246082309ed.slice/crio-66fe5f1163e11490ddc3b12e8b040e4e3699ff97d3dd9519a5baa16b00be6c27 WatchSource:0}: Error finding container 66fe5f1163e11490ddc3b12e8b040e4e3699ff97d3dd9519a5baa16b00be6c27: Status 404 returned error can't find the container with id 66fe5f1163e11490ddc3b12e8b040e4e3699ff97d3dd9519a5baa16b00be6c27 Jul 28 12:04:13.164620 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:13.164596 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf368fa_46b2_4bbc_a2a0_d3f35022719a.slice/crio-65fbb90cf9024d2b2470a67b3b6139338f07847d3fcb0b07bcae57c4d56322c1 WatchSource:0}: Error finding container 65fbb90cf9024d2b2470a67b3b6139338f07847d3fcb0b07bcae57c4d56322c1: Status 404 returned error can't find the container with id 65fbb90cf9024d2b2470a67b3b6139338f07847d3fcb0b07bcae57c4d56322c1 Jul 28 12:04:13.165405 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:13.165384 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12686a8e_d528_4138_a8ff_0eaf60244bd2.slice/crio-6486c54437e448d49f3f94d9cfa6a26a9ddc779efa982667f3acce1adac33219 WatchSource:0}: Error finding container 6486c54437e448d49f3f94d9cfa6a26a9ddc779efa982667f3acce1adac33219: Status 404 returned error can't find the container with id 6486c54437e448d49f3f94d9cfa6a26a9ddc779efa982667f3acce1adac33219 Jul 28 12:04:14.089678 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:14.089589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" event={"ID":"cd4840bc-966d-4609-b62f-2246082309ed","Type":"ContainerStarted","Data":"66fe5f1163e11490ddc3b12e8b040e4e3699ff97d3dd9519a5baa16b00be6c27"} Jul 28 12:04:14.091320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:14.091293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4mk2d" event={"ID":"caf368fa-46b2-4bbc-a2a0-d3f35022719a","Type":"ContainerStarted","Data":"65fbb90cf9024d2b2470a67b3b6139338f07847d3fcb0b07bcae57c4d56322c1"} Jul 28 12:04:14.094073 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:14.094047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trjwp" event={"ID":"aec5250b-4ff2-414b-aab2-a8c726a09765","Type":"ContainerStarted","Data":"568105eff9c090efb56f5906822bda8eb322681feed57485171f0dd6cecbb80e"} Jul 28 12:04:14.097142 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:14.097120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hmlhv" event={"ID":"12686a8e-d528-4138-a8ff-0eaf60244bd2","Type":"ContainerStarted","Data":"6486c54437e448d49f3f94d9cfa6a26a9ddc779efa982667f3acce1adac33219"} Jul 28 12:04:14.098696 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:14.098674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gklcw" event={"ID":"0afc5b7d-6f03-4e22-b639-88fa243b94d3","Type":"ContainerStarted","Data":"81822aab92c3ad6ae4d0f4e22710a458d15838f877909841f978cdb5e2707b84"} Jul 28 12:04:14.126240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:14.126182 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-trjwp" podStartSLOduration=2.304692969 podStartE2EDuration="17.126156499s" totalImagesPullingTime="14.82146353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:14.125791914 +0000 UTC m=+67.942721732" watchObservedRunningTime="2026-07-28 12:04:14.126156499 +0000 UTC m=+67.943086317" Jul 28 12:04:15.107923 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:15.107808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" event={"ID":"cd4840bc-966d-4609-b62f-2246082309ed","Type":"ContainerStarted","Data":"2689cb9d62c77810af7f0cbb225e7f1580be5ebb9aaff2c123c72d866e71bfd2"} Jul 28 12:04:15.108292 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:15.107940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" event={"ID":"cd4840bc-966d-4609-b62f-2246082309ed","Type":"ContainerStarted","Data":"5ceb6bf0c8b6eaa8d966f2bf0cca9910d5ea0b717fda82c02ccc3e5c04861179"} Jul 28 12:04:15.111930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:15.111899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gklcw" event={"ID":"0afc5b7d-6f03-4e22-b639-88fa243b94d3","Type":"ContainerStarted","Data":"641a7b765fcd1011a56b79788a4d8db376b38c1b889c9efb82140307d76ea78f"} Jul 28 12:04:15.112095 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:15.111954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gklcw" event={"ID":"0afc5b7d-6f03-4e22-b639-88fa243b94d3","Type":"ContainerStarted","Data":"e0382b2a3b23dd6ee492a5a5f339551e86dd5c60e45d1fa80f4c93257ec025ef"} Jul 28 12:04:15.167485 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:15.166325 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-f2vhq" podStartSLOduration=3.596184431 podStartE2EDuration="5.166301579s" totalImagesPullingTime="1.570117148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:15.164522361 +0000 UTC m=+68.981452190" watchObservedRunningTime="2026-07-28 12:04:15.166301579 +0000 UTC m=+68.983231397" Jul 28 12:04:15.220286 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:15.219354 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gklcw" podStartSLOduration=67.133734476 podStartE2EDuration="1m8.219334271s" totalImagesPullingTime="1.085599795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:15.218423312 +0000 UTC m=+69.035353131" watchObservedRunningTime="2026-07-28 12:04:15.219334271 +0000 UTC m=+69.036264089" Jul 28 12:04:16.048348 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:16.048314 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kr5xt" Jul 28 12:04:18.126865 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:18.126829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hmlhv" event={"ID":"12686a8e-d528-4138-a8ff-0eaf60244bd2","Type":"ContainerStarted","Data":"33720911914cefd0e9b9278183e069df64701eae5243151098d20dbc761c7124"} Jul 28 12:04:18.127285 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:18.126948 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:04:18.128262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:18.128237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4mk2d" event={"ID":"caf368fa-46b2-4bbc-a2a0-d3f35022719a","Type":"ContainerStarted","Data":"f2c6fd15be0601840c92eadb85581de649258e493c403a98263d96f7ef0e3e7b"} Jul 28 12:04:18.157772 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:18.157688 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hmlhv" podStartSLOduration=66.447669986 podStartE2EDuration="1m11.157675635s" totalImagesPullingTime="4.710005649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:18.15598529 +0000 UTC m=+71.972915107" watchObservedRunningTime="2026-07-28 12:04:18.157675635 +0000 UTC m=+71.974605452" Jul 28 12:04:19.004414 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:19.004374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:19.004585 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:19.004442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:19.009742 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:19.009717 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:19.035618 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:19.035556 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4mk2d" podStartSLOduration=67.320965352 podStartE2EDuration="1m12.035540992s" totalImagesPullingTime="4.71457564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:18.177441617 +0000 UTC m=+71.994371434" watchObservedRunningTime="2026-07-28 12:04:19.035540992 +0000 UTC m=+72.852470809" Jul 28 12:04:19.135986 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:19.135961 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:04:19.221083 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:19.221052 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df698df75-7sp9r"] Jul 28 12:04:21.422094 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.422059 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b"] Jul 28 12:04:21.424200 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.424184 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.431203 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.431184 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dzcz7\"" Jul 28 12:04:21.431337 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.431243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 12:04:21.431556 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.431539 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 12:04:21.445712 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.445680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b"] Jul 28 12:04:21.526167 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.526136 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-278xn"] Jul 28 12:04:21.528247 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.528213 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.533651 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.533622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 12:04:21.533786 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.533681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 12:04:21.533940 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.533920 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mk8bl\"" Jul 28 12:04:21.534014 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.534001 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 12:04:21.550139 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.550117 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gtw7p"] Jul 28 12:04:21.552229 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.552207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.555086 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.555065 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-278xn"] Jul 28 12:04:21.559102 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.558793 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 12:04:21.559347 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.559329 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 12:04:21.559429 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.559376 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 12:04:21.563812 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.563798 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vfmx2\"" Jul 28 12:04:21.569288 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.569263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxxst\" (UniqueName: \"kubernetes.io/projected/c351e7c4-094e-4393-a391-0fcb3e67ed1d-kube-api-access-mxxst\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.569350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.569300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.569350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.569321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.569416 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.569363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c351e7c4-094e-4393-a391-0fcb3e67ed1d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.670151 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670120 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.670151 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670393 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c351e7c4-094e-4393-a391-0fcb3e67ed1d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.670393 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:21.670271 2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 28 12:04:21.670393 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.670393 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:21.670344 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-tls podName:c351e7c4-094e-4393-a391-0fcb3e67ed1d nodeName:}" failed. No retries permitted until 2026-07-28 12:04:22.170323319 +0000 UTC m=+75.987253118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-vfd2b" (UID: "c351e7c4-094e-4393-a391-0fcb3e67ed1d") : secret "openshift-state-metrics-tls" not found Jul 28 12:04:21.670393 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-root\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-wtmp\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.670606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-sys\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a0f45c-2485-44cc-beae-35eb28bd86d1-metrics-client-ca\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.670815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.670815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-tls\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-textfile\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.670815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.670815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxst\" (UniqueName: \"kubernetes.io/projected/c351e7c4-094e-4393-a391-0fcb3e67ed1d-kube-api-access-mxxst\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.670815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.670815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsk5\" (UniqueName: \"kubernetes.io/projected/d1a0f45c-2485-44cc-beae-35eb28bd86d1-kube-api-access-bwsk5\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.671058 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44sq\" (UniqueName: \"kubernetes.io/projected/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-api-access-g44sq\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.671058 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.670845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c351e7c4-094e-4393-a391-0fcb3e67ed1d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.673181 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.673129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.692474 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.692446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxxst\" (UniqueName: \"kubernetes.io/projected/c351e7c4-094e-4393-a391-0fcb3e67ed1d-kube-api-access-mxxst\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:21.772033 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-tls\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772216 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-textfile\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772216 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.772216 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsk5\" (UniqueName: \"kubernetes.io/projected/d1a0f45c-2485-44cc-beae-35eb28bd86d1-kube-api-access-bwsk5\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772216 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44sq\" (UniqueName: \"kubernetes.io/projected/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-api-access-g44sq\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.772216 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:21.772158 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 12:04:21.772451 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:21.772222 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-tls podName:d1a0f45c-2485-44cc-beae-35eb28bd86d1 nodeName:}" failed. No retries permitted until 2026-07-28 12:04:22.272201686 +0000 UTC m=+76.089131498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-tls") pod "node-exporter-gtw7p" (UID: "d1a0f45c-2485-44cc-beae-35eb28bd86d1") : secret "node-exporter-tls" not found Jul 28 12:04:21.772451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.772451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-root\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-textfile\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772405 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-root\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-wtmp\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772576 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-wtmp\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-sys\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a0f45c-2485-44cc-beae-35eb28bd86d1-metrics-client-ca\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1a0f45c-2485-44cc-beae-35eb28bd86d1-sys\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.772791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.773423 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.772819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.773423 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.773139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.773423 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.773232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a0f45c-2485-44cc-beae-35eb28bd86d1-metrics-client-ca\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.773423 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.773252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.773652 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.773628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.774947 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.774924 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.775160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.775144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.775214 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.775178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.787699 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.787677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsk5\" (UniqueName: \"kubernetes.io/projected/d1a0f45c-2485-44cc-beae-35eb28bd86d1-kube-api-access-bwsk5\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:21.788338 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.788319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44sq\" (UniqueName: \"kubernetes.io/projected/5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9-kube-api-access-g44sq\") pod \"kube-state-metrics-56d9f944b4-278xn\" (UID: \"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:21.836750 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:21.836717 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" Jul 28 12:04:22.071780 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.071758 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-278xn"] Jul 28 12:04:22.074170 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:22.074145 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cea282d_37c7_4a3a_a1ae_23e65b1bdeb9.slice/crio-ec3ac3d2dc504d91d4cf70cae1b00769862686312214dedabb2682f3c06af8fe WatchSource:0}: Error finding container ec3ac3d2dc504d91d4cf70cae1b00769862686312214dedabb2682f3c06af8fe: Status 404 returned error can't find the container with id ec3ac3d2dc504d91d4cf70cae1b00769862686312214dedabb2682f3c06af8fe Jul 28 12:04:22.144478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.144442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" event={"ID":"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9","Type":"ContainerStarted","Data":"ec3ac3d2dc504d91d4cf70cae1b00769862686312214dedabb2682f3c06af8fe"} Jul 28 12:04:22.176028 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.176004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:22.178592 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.178574 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c351e7c4-094e-4393-a391-0fcb3e67ed1d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-vfd2b\" (UID: \"c351e7c4-094e-4393-a391-0fcb3e67ed1d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:22.276927 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.276862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-tls\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:22.279268 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.279251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1a0f45c-2485-44cc-beae-35eb28bd86d1-node-exporter-tls\") pod \"node-exporter-gtw7p\" (UID: \"d1a0f45c-2485-44cc-beae-35eb28bd86d1\") " pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:22.332385 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.332309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" Jul 28 12:04:22.462016 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.461978 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtw7p" Jul 28 12:04:22.471761 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:22.471737 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a0f45c_2485_44cc_beae_35eb28bd86d1.slice/crio-82502de1110a4d8bb6ea5f4170e645467c91ea48113b4ef3abc941e20ddf58d3 WatchSource:0}: Error finding container 82502de1110a4d8bb6ea5f4170e645467c91ea48113b4ef3abc941e20ddf58d3: Status 404 returned error can't find the container with id 82502de1110a4d8bb6ea5f4170e645467c91ea48113b4ef3abc941e20ddf58d3 Jul 28 12:04:22.476846 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.476826 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b"] Jul 28 12:04:22.479342 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:22.479315 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc351e7c4_094e_4393_a391_0fcb3e67ed1d.slice/crio-ab822be89ba844be112e196f23b3457f88856364b2565bd2a9f8be2195fee1f1 WatchSource:0}: Error finding container ab822be89ba844be112e196f23b3457f88856364b2565bd2a9f8be2195fee1f1: Status 404 returned error can't find the container with id ab822be89ba844be112e196f23b3457f88856364b2565bd2a9f8be2195fee1f1 Jul 28 12:04:22.728241 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.726852 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 12:04:22.729439 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.729375 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.732122 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.732102 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 12:04:22.739901 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.739863 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 12:04:22.741776 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.741467 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 12:04:22.741891 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.741516 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 12:04:22.762784 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.762718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 12:04:22.763053 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.762795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 12:04:22.763177 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.762853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nw27l\"" Jul 28 12:04:22.763177 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.762960 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 12:04:22.763463 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.763443 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 12:04:22.765542 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.765512 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 12:04:22.767657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.767490 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 12:04:22.879722 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.879722 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.879722 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efd872fc-5cef-4712-b468-3ea71bcd14f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efd872fc-5cef-4712-b468-3ea71bcd14f2-config-out\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efd872fc-5cef-4712-b468-3ea71bcd14f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efd872fc-5cef-4712-b468-3ea71bcd14f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxgt\" (UniqueName: \"kubernetes.io/projected/efd872fc-5cef-4712-b468-3ea71bcd14f2-kube-api-access-4fxgt\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.879969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880263 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.880015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-web-config\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880263 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.880037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efd872fc-5cef-4712-b468-3ea71bcd14f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.880263 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.880065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.980985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-web-config\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efd872fc-5cef-4712-b468-3ea71bcd14f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efd872fc-5cef-4712-b468-3ea71bcd14f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efd872fc-5cef-4712-b468-3ea71bcd14f2-config-out\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efd872fc-5cef-4712-b468-3ea71bcd14f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efd872fc-5cef-4712-b468-3ea71bcd14f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxgt\" (UniqueName: \"kubernetes.io/projected/efd872fc-5cef-4712-b468-3ea71bcd14f2-kube-api-access-4fxgt\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.981740 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.981327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.983006 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.982983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/efd872fc-5cef-4712-b468-3ea71bcd14f2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.985180 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.985148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efd872fc-5cef-4712-b468-3ea71bcd14f2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.985485 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.985457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efd872fc-5cef-4712-b468-3ea71bcd14f2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.988964 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.986458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.988964 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.987766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.989546 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.989523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efd872fc-5cef-4712-b468-3ea71bcd14f2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.990065 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.990044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.990180 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.990159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.990251 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.990232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-config-volume\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.990666 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.990647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-web-config\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.991065 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.991045 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/efd872fc-5cef-4712-b468-3ea71bcd14f2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.991542 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.991517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efd872fc-5cef-4712-b468-3ea71bcd14f2-config-out\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:22.995865 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:22.995839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxgt\" (UniqueName: \"kubernetes.io/projected/efd872fc-5cef-4712-b468-3ea71bcd14f2-kube-api-access-4fxgt\") pod \"alertmanager-main-0\" (UID: \"efd872fc-5cef-4712-b468-3ea71bcd14f2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:23.043022 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:23.042983 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 12:04:23.151686 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:23.151651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtw7p" event={"ID":"d1a0f45c-2485-44cc-beae-35eb28bd86d1","Type":"ContainerStarted","Data":"82502de1110a4d8bb6ea5f4170e645467c91ea48113b4ef3abc941e20ddf58d3"} Jul 28 12:04:23.155210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:23.154817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" event={"ID":"c351e7c4-094e-4393-a391-0fcb3e67ed1d","Type":"ContainerStarted","Data":"30864da647044bc8308d44a02cdd381feeee23aa7a294ef8d74b4d1432fb9fb4"} Jul 28 12:04:23.155210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:23.154900 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" event={"ID":"c351e7c4-094e-4393-a391-0fcb3e67ed1d","Type":"ContainerStarted","Data":"12779e85068524f050fa191941dfab7bd9b56cc081e65c0e32c72afbb4c3efbd"} Jul 28 12:04:23.155210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:23.154936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" event={"ID":"c351e7c4-094e-4393-a391-0fcb3e67ed1d","Type":"ContainerStarted","Data":"ab822be89ba844be112e196f23b3457f88856364b2565bd2a9f8be2195fee1f1"} Jul 28 12:04:23.212182 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:23.212152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 12:04:23.366662 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:23.366569 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd872fc_5cef_4712_b468_3ea71bcd14f2.slice/crio-e64a3b27a8ab08f9ff0b5fbdcb354c28a1e61688d478b8c0874c8bc6fec4facc WatchSource:0}: Error finding container e64a3b27a8ab08f9ff0b5fbdcb354c28a1e61688d478b8c0874c8bc6fec4facc: Status 404 returned error can't find the container with id e64a3b27a8ab08f9ff0b5fbdcb354c28a1e61688d478b8c0874c8bc6fec4facc Jul 28 12:04:24.162155 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.162115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerStarted","Data":"e64a3b27a8ab08f9ff0b5fbdcb354c28a1e61688d478b8c0874c8bc6fec4facc"} Jul 28 12:04:24.164871 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.164842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" event={"ID":"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9","Type":"ContainerStarted","Data":"4e253bc735786dc12834fc7de8d5b67df04baa917f21f7ecd2bc0860f578c23a"} Jul 28 12:04:24.165013 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.164953 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" event={"ID":"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9","Type":"ContainerStarted","Data":"19229c5c0c72cf3dae4ed58685ca5ec5364236ab1fe7493fea7cfb8b1f5ef4a0"} Jul 28 12:04:24.167002 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.166974 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1a0f45c-2485-44cc-beae-35eb28bd86d1" containerID="7ecd55292cf29bbfb1f4c7ab0ffaa690f9c14ea8567a2600a10061ef17dc21c5" exitCode=0 Jul 28 12:04:24.167109 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.167052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtw7p" event={"ID":"d1a0f45c-2485-44cc-beae-35eb28bd86d1","Type":"ContainerDied","Data":"7ecd55292cf29bbfb1f4c7ab0ffaa690f9c14ea8567a2600a10061ef17dc21c5"} Jul 28 12:04:24.170243 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.170204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" event={"ID":"c351e7c4-094e-4393-a391-0fcb3e67ed1d","Type":"ContainerStarted","Data":"1eda904a30556862468ab45b037feb957f01b3d40bd0d69b9c1789f05143ff8a"} Jul 28 12:04:24.615895 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.615714 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-vfd2b" podStartSLOduration=2.25969192 podStartE2EDuration="3.61568917s" totalImagesPullingTime="1.35599725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:24.239676121 +0000 UTC m=+78.056605939" watchObservedRunningTime="2026-07-28 12:04:24.61568917 +0000 UTC m=+78.432618988" Jul 28 12:04:24.617466 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.616899 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c556dff8f-52c5j"] Jul 28 12:04:24.620948 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.620920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.624588 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.624564 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 12:04:24.624734 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.624722 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 12:04:24.624812 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.624779 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lmh5l\"" Jul 28 12:04:24.625007 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.624984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 12:04:24.625508 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.625266 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d1cvtroptshke\"" Jul 28 12:04:24.625508 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.625282 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 12:04:24.625508 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.625451 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 12:04:24.669802 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.669772 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c556dff8f-52c5j"] Jul 28 12:04:24.698096 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-tls\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.698208 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.698208 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-grpc-tls\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.698304 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.698304 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/179ffc7c-5199-4cc9-9a62-64678a1f4449-metrics-client-ca\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.698304 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.698407 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.698407 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.698353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvb6l\" (UniqueName: \"kubernetes.io/projected/179ffc7c-5199-4cc9-9a62-64678a1f4449-kube-api-access-bvb6l\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.798922 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.798868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvb6l\" (UniqueName: \"kubernetes.io/projected/179ffc7c-5199-4cc9-9a62-64678a1f4449-kube-api-access-bvb6l\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.799115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.798945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-tls\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.799115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.798964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.799115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.798983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-grpc-tls\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.799115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.799014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.799115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.799065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/179ffc7c-5199-4cc9-9a62-64678a1f4449-metrics-client-ca\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.799115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.799093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.799115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.799117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.802160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.802126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/179ffc7c-5199-4cc9-9a62-64678a1f4449-metrics-client-ca\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.805235 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.805204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.806621 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.806593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.807347 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.807297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.807504 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.807479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-tls\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.807573 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.807486 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.807622 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.807587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/179ffc7c-5199-4cc9-9a62-64678a1f4449-secret-grpc-tls\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.810018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.809994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvb6l\" (UniqueName: \"kubernetes.io/projected/179ffc7c-5199-4cc9-9a62-64678a1f4449-kube-api-access-bvb6l\") pod \"thanos-querier-5c556dff8f-52c5j\" (UID: \"179ffc7c-5199-4cc9-9a62-64678a1f4449\") " pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:24.931379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:24.931358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:25.092157 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.092129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c556dff8f-52c5j"] Jul 28 12:04:25.096530 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:25.096503 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179ffc7c_5199_4cc9_9a62_64678a1f4449.slice/crio-2355d6f344e5672fd7b82262f7f000fa428cb37f14388145e6ba1c74bcd978f2 WatchSource:0}: Error finding container 2355d6f344e5672fd7b82262f7f000fa428cb37f14388145e6ba1c74bcd978f2: Status 404 returned error can't find the container with id 2355d6f344e5672fd7b82262f7f000fa428cb37f14388145e6ba1c74bcd978f2 Jul 28 12:04:25.176304 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.176214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" event={"ID":"5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9","Type":"ContainerStarted","Data":"11571ec7a4e65004c03ff46112dbd2b843a753a6e0b1302e4da4d6420c60acd9"} Jul 28 12:04:25.178513 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.178486 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtw7p" event={"ID":"d1a0f45c-2485-44cc-beae-35eb28bd86d1","Type":"ContainerStarted","Data":"b63f02e55bccfcc2df73ce43e5e45e12dfd11b0c2fc01bc849a313bad5260ee1"} Jul 28 12:04:25.178639 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.178535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtw7p" event={"ID":"d1a0f45c-2485-44cc-beae-35eb28bd86d1","Type":"ContainerStarted","Data":"7e74aeb9413b25911ef8bf47c90294c4d44d02ebaf9ba08754b772dcfcdd7e2a"} Jul 28 12:04:25.180066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.180042 2564 generic.go:356] "Generic (PLEG): container finished" podID="efd872fc-5cef-4712-b468-3ea71bcd14f2" containerID="70d70c45e95ecfbcad15775cd7ab97e52e6a0ee400740619322a689ca52253f4" exitCode=0 Jul 28 12:04:25.180174 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.180117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerDied","Data":"70d70c45e95ecfbcad15775cd7ab97e52e6a0ee400740619322a689ca52253f4"} Jul 28 12:04:25.181431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.181404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" event={"ID":"179ffc7c-5199-4cc9-9a62-64678a1f4449","Type":"ContainerStarted","Data":"2355d6f344e5672fd7b82262f7f000fa428cb37f14388145e6ba1c74bcd978f2"} Jul 28 12:04:25.203723 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:25.203677 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-278xn" podStartSLOduration=2.315349045 podStartE2EDuration="4.203662251s" totalImagesPullingTime="1.888313206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:25.202763565 +0000 UTC m=+79.019693382" watchObservedRunningTime="2026-07-28 12:04:25.203662251 +0000 UTC m=+79.020592068" Jul 28 12:04:26.181023 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.180965 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gtw7p" podStartSLOduration=3.604301914 podStartE2EDuration="5.180945299s" totalImagesPullingTime="1.490846333s" totalInitContainerRuntime="85.797052ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:25.313304713 +0000 UTC m=+79.130234530" watchObservedRunningTime="2026-07-28 12:04:26.180945299 +0000 UTC m=+79.997875158" Jul 28 12:04:26.181401 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.181260 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57ff94bd4-krfz9"] Jul 28 12:04:26.185125 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.184691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.224655 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.224424 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57ff94bd4-krfz9"] Jul 28 12:04:26.227263 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.225987 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4"] Jul 28 12:04:26.229407 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.229389 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:26.236207 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.236182 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zbddq\"" Jul 28 12:04:26.236421 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.236402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 12:04:26.250279 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.250255 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4"] Jul 28 12:04:26.285822 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.285794 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-768c457df4-csh9b"] Jul 28 12:04:26.289075 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.289054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.295566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.295123 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 12:04:26.295566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.295129 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 12:04:26.295566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.295258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 12:04:26.295566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.295375 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 12:04:26.295566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.295452 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-k6vgdivo5tc0\"" Jul 28 12:04:26.295566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.295472 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2jk6h\"" Jul 28 12:04:26.310711 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.310671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a90f56d7-67cd-4a18-8464-95f197f4d8c3-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-mv4k4\" (UID: \"a90f56d7-67cd-4a18-8464-95f197f4d8c3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:26.310917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.310737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-serving-cert\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.310917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.310776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-oauth-config\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.310917 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.310798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-oauth-serving-cert\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.311350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.310923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2br8k\" (UniqueName: \"kubernetes.io/projected/a90f56d7-67cd-4a18-8464-95f197f4d8c3-kube-api-access-2br8k\") pod \"monitoring-plugin-68b7fc9574-mv4k4\" (UID: \"a90f56d7-67cd-4a18-8464-95f197f4d8c3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:26.311350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.310985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqz2\" (UniqueName: \"kubernetes.io/projected/6ef4d210-7700-4a6a-8c63-f734c6dd4add-kube-api-access-hbqz2\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.311350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.311012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-trusted-ca-bundle\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.311350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.311137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-service-ca\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.311350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.311175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-config\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.318653 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.318625 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-768c457df4-csh9b"] Jul 28 12:04:26.412594 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.412558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-serving-cert\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.412594 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.412606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-secret-metrics-server-client-certs\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.412955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.412632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-oauth-config\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.412955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.412658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-oauth-serving-cert\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.412955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.412811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ec989cab-4f7d-459a-8002-38978bd5636a-metrics-server-audit-profiles\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.412955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.412852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2br8k\" (UniqueName: \"kubernetes.io/projected/a90f56d7-67cd-4a18-8464-95f197f4d8c3-kube-api-access-2br8k\") pod \"monitoring-plugin-68b7fc9574-mv4k4\" (UID: \"a90f56d7-67cd-4a18-8464-95f197f4d8c3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:26.412955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.412906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ec989cab-4f7d-459a-8002-38978bd5636a-audit-log\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.413218 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqz2\" (UniqueName: \"kubernetes.io/projected/6ef4d210-7700-4a6a-8c63-f734c6dd4add-kube-api-access-hbqz2\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.413218 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-trusted-ca-bundle\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.413218 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshtz\" (UniqueName: \"kubernetes.io/projected/ec989cab-4f7d-459a-8002-38978bd5636a-kube-api-access-cshtz\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.413218 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-client-ca-bundle\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.413453 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-service-ca\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.413453 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-config\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.413453 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-secret-metrics-server-tls\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.413453 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec989cab-4f7d-459a-8002-38978bd5636a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.413777 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a90f56d7-67cd-4a18-8464-95f197f4d8c3-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-mv4k4\" (UID: \"a90f56d7-67cd-4a18-8464-95f197f4d8c3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:26.413777 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.413625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-oauth-serving-cert\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.413777 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:26.413683 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 28 12:04:26.413777 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:26.413756 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a90f56d7-67cd-4a18-8464-95f197f4d8c3-monitoring-plugin-cert podName:a90f56d7-67cd-4a18-8464-95f197f4d8c3 nodeName:}" failed. No retries permitted until 2026-07-28 12:04:26.913736888 +0000 UTC m=+80.730666683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a90f56d7-67cd-4a18-8464-95f197f4d8c3-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-mv4k4" (UID: "a90f56d7-67cd-4a18-8464-95f197f4d8c3") : secret "monitoring-plugin-cert" not found Jul 28 12:04:26.414088 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.414069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-trusted-ca-bundle\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.414544 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.414515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-config\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.414669 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.414610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-service-ca\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.415600 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.415580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-oauth-config\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.415686 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.415583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-serving-cert\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.436524 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.436460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqz2\" (UniqueName: \"kubernetes.io/projected/6ef4d210-7700-4a6a-8c63-f734c6dd4add-kube-api-access-hbqz2\") pod \"console-57ff94bd4-krfz9\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.436993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.436969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2br8k\" (UniqueName: \"kubernetes.io/projected/a90f56d7-67cd-4a18-8464-95f197f4d8c3-kube-api-access-2br8k\") pod \"monitoring-plugin-68b7fc9574-mv4k4\" (UID: \"a90f56d7-67cd-4a18-8464-95f197f4d8c3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:26.497972 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.497947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:26.514826 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.514790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-secret-metrics-server-tls\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515024 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.514843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec989cab-4f7d-459a-8002-38978bd5636a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515024 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.514906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-secret-metrics-server-client-certs\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515143 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.515021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ec989cab-4f7d-459a-8002-38978bd5636a-metrics-server-audit-profiles\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515143 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.515082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ec989cab-4f7d-459a-8002-38978bd5636a-audit-log\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515143 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.515115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshtz\" (UniqueName: \"kubernetes.io/projected/ec989cab-4f7d-459a-8002-38978bd5636a-kube-api-access-cshtz\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515277 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.515162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-client-ca-bundle\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515675 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.515611 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ec989cab-4f7d-459a-8002-38978bd5636a-audit-log\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.515783 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.515676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec989cab-4f7d-459a-8002-38978bd5636a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.516336 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.516310 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ec989cab-4f7d-459a-8002-38978bd5636a-metrics-server-audit-profiles\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.518002 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.517983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-client-ca-bundle\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.518138 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.518120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-secret-metrics-server-client-certs\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.518309 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.518291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ec989cab-4f7d-459a-8002-38978bd5636a-secret-metrics-server-tls\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.526411 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.526383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshtz\" (UniqueName: \"kubernetes.io/projected/ec989cab-4f7d-459a-8002-38978bd5636a-kube-api-access-cshtz\") pod \"metrics-server-768c457df4-csh9b\" (UID: \"ec989cab-4f7d-459a-8002-38978bd5636a\") " pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.600805 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.600775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:26.784020 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.783996 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b4996fbf9-dmx8d"] Jul 28 12:04:26.787593 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.787572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.792518 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.792493 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 12:04:26.793503 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.793482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 12:04:26.793611 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.793537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 12:04:26.793611 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.793537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 12:04:26.793731 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.793535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 12:04:26.795063 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.795043 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-m4dw8\"" Jul 28 12:04:26.808042 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.808013 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 12:04:26.813336 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.813307 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b4996fbf9-dmx8d"] Jul 28 12:04:26.918474 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-serving-certs-ca-bundle\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.918474 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-telemeter-client-tls\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.918697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a90f56d7-67cd-4a18-8464-95f197f4d8c3-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-mv4k4\" (UID: \"a90f56d7-67cd-4a18-8464-95f197f4d8c3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:26.918697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-metrics-client-ca\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.918697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-secret-telemeter-client\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.918697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-federate-client-tls\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.918928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdch\" (UniqueName: \"kubernetes.io/projected/e64e69a2-05f2-43a3-a02b-226b17f00515-kube-api-access-rvdch\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.918928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.918928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.918835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:26.921410 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:26.921392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a90f56d7-67cd-4a18-8464-95f197f4d8c3-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-mv4k4\" (UID: \"a90f56d7-67cd-4a18-8464-95f197f4d8c3\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:27.019639 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.019598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-telemeter-client-tls\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.019828 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.019678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-metrics-client-ca\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.019828 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.019724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-secret-telemeter-client\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.019828 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.019750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-federate-client-tls\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.019828 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.019773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdch\" (UniqueName: \"kubernetes.io/projected/e64e69a2-05f2-43a3-a02b-226b17f00515-kube-api-access-rvdch\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.020054 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.020027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.021070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.020094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.021070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.020156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-serving-certs-ca-bundle\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.021070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.020800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-metrics-client-ca\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.021070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.020827 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-serving-certs-ca-bundle\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.021070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.020976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64e69a2-05f2-43a3-a02b-226b17f00515-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.022723 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.022678 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-telemeter-client-tls\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.023159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.023135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-secret-telemeter-client\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.023462 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.023440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-federate-client-tls\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.026420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.023988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e64e69a2-05f2-43a3-a02b-226b17f00515-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.031385 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.031362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdch\" (UniqueName: \"kubernetes.io/projected/e64e69a2-05f2-43a3-a02b-226b17f00515-kube-api-access-rvdch\") pod \"telemeter-client-b4996fbf9-dmx8d\" (UID: \"e64e69a2-05f2-43a3-a02b-226b17f00515\") " pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.101754 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.101626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" Jul 28 12:04:27.142365 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.142266 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:27.537437 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.537408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-768c457df4-csh9b"] Jul 28 12:04:27.552110 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:27.552079 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec989cab_4f7d_459a_8002_38978bd5636a.slice/crio-90cc01f1e5716e27bc7b37657aeecce89ce3c3cc0e58cde18525dbe638212f03 WatchSource:0}: Error finding container 90cc01f1e5716e27bc7b37657aeecce89ce3c3cc0e58cde18525dbe638212f03: Status 404 returned error can't find the container with id 90cc01f1e5716e27bc7b37657aeecce89ce3c3cc0e58cde18525dbe638212f03 Jul 28 12:04:27.786213 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.786188 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57ff94bd4-krfz9"] Jul 28 12:04:27.789021 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:27.788990 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef4d210_7700_4a6a_8c63_f734c6dd4add.slice/crio-f9c3536a5ec0c2ecc3c48fbf66a568e44ba509e8236daf72d5d80f1b9521dddc WatchSource:0}: Error finding container f9c3536a5ec0c2ecc3c48fbf66a568e44ba509e8236daf72d5d80f1b9521dddc: Status 404 returned error can't find the container with id f9c3536a5ec0c2ecc3c48fbf66a568e44ba509e8236daf72d5d80f1b9521dddc Jul 28 12:04:27.809782 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.809678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b4996fbf9-dmx8d"] Jul 28 12:04:27.811379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.811356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4"] Jul 28 12:04:27.812321 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:27.812183 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64e69a2_05f2_43a3_a02b_226b17f00515.slice/crio-fa3d2fb0ae22eb7c02db16840e6aaafdf5aa449f0435d50c1ed2bd22b309205c WatchSource:0}: Error finding container fa3d2fb0ae22eb7c02db16840e6aaafdf5aa449f0435d50c1ed2bd22b309205c: Status 404 returned error can't find the container with id fa3d2fb0ae22eb7c02db16840e6aaafdf5aa449f0435d50c1ed2bd22b309205c Jul 28 12:04:27.812986 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:04:27.812798 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90f56d7_67cd_4a18_8464_95f197f4d8c3.slice/crio-cbaea465aec38ba947e22584bb08e6eba5770eb96bb7e87d1c83dd4ef335d401 WatchSource:0}: Error finding container cbaea465aec38ba947e22584bb08e6eba5770eb96bb7e87d1c83dd4ef335d401: Status 404 returned error can't find the container with id cbaea465aec38ba947e22584bb08e6eba5770eb96bb7e87d1c83dd4ef335d401 Jul 28 12:04:27.941994 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.941619 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 12:04:27.946257 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.946233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:27.951543 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.951500 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wf7cx\"" Jul 28 12:04:27.951694 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.951619 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 28 12:04:27.952398 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.952321 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 28 12:04:27.952398 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.952354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 28 12:04:27.952555 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.952412 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 28 12:04:27.952555 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.952531 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 28 12:04:27.952555 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.952535 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 28 12:04:27.952713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.952683 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 28 12:04:27.952981 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.952964 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 28 12:04:27.955566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.955413 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 28 12:04:27.956988 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.956707 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ecafe2kd06bi9\"" Jul 28 12:04:27.956988 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.956825 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 28 12:04:27.956988 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.956863 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 28 12:04:27.957545 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.957412 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 28 12:04:27.958082 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.958063 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 28 12:04:27.964500 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.964473 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 28 12:04:27.990183 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:27.990152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 12:04:28.031000 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.030956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp2xg\" (UniqueName: \"kubernetes.io/projected/95ac02ce-43e4-4f24-9bad-f5936f09028c-kube-api-access-vp2xg\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031471 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031471 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031471 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031471 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95ac02ce-43e4-4f24-9bad-f5936f09028c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031683 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031683 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031683 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031683 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95ac02ce-43e4-4f24-9bad-f5936f09028c-config-out\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031683 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031949 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.031949 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-config\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.032382 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.031770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-web-config\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.032799 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.032769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.032934 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.032916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139612 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139612 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139612 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp2xg\" (UniqueName: \"kubernetes.io/projected/95ac02ce-43e4-4f24-9bad-f5936f09028c-kube-api-access-vp2xg\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.139900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95ac02ce-43e4-4f24-9bad-f5936f09028c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.139999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95ac02ce-43e4-4f24-9bad-f5936f09028c-config-out\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-config\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-web-config\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140270 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140755 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140755 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140498 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140755 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.140755 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.140655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.141504 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.141186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.141627 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.141548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.142964 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.142586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.144930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.144905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95ac02ce-43e4-4f24-9bad-f5936f09028c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.146263 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.146236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95ac02ce-43e4-4f24-9bad-f5936f09028c-config-out\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.148571 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.148547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150065 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.149626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-web-config\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150065 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.150039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150065 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.150040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150249 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.150122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95ac02ce-43e4-4f24-9bad-f5936f09028c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150249 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.150164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150249 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.150218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150576 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.150538 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.150675 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.150616 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.152249 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.152208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/95ac02ce-43e4-4f24-9bad-f5936f09028c-config\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.153374 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.153334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp2xg\" (UniqueName: \"kubernetes.io/projected/95ac02ce-43e4-4f24-9bad-f5936f09028c-kube-api-access-vp2xg\") pod \"prometheus-k8s-0\" (UID: \"95ac02ce-43e4-4f24-9bad-f5936f09028c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.204381 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.204334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerStarted","Data":"3251e2e905fda69c1d17fe22fea461e6cc08ef3d7dd4bb5829b69492afef02b4"} Jul 28 12:04:28.204563 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.204470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerStarted","Data":"7b0e8e9c80b7f6fd0071dbc595e76c76191ecb83e60188960b98d06fde9c1cc9"} Jul 28 12:04:28.204563 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.204511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerStarted","Data":"3eff92a55ac4c7e4ce6a099556079b53de448c1099663ff9ac14b72bde79cff4"} Jul 28 12:04:28.204563 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.204549 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerStarted","Data":"3f5f0e82ab8f91842e7a3d40700d689cd05cee70b5be669b6d2bb8cb445b9caf"} Jul 28 12:04:28.204729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.204587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerStarted","Data":"77aff87eae2ec2e99e1e3122e268c5ca278c26dc06db0002548dd99c3670bc17"} Jul 28 12:04:28.209152 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.209110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" event={"ID":"179ffc7c-5199-4cc9-9a62-64678a1f4449","Type":"ContainerStarted","Data":"957a0e3eac2a621b69ede9f025901da8bd7898c11e1d59e657e25448aa94c9b9"} Jul 28 12:04:28.209340 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.209201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" event={"ID":"179ffc7c-5199-4cc9-9a62-64678a1f4449","Type":"ContainerStarted","Data":"4e38d01ecaa0df3a73984ec59b6cfca932f6fe2930e74d065118a013e512c771"} Jul 28 12:04:28.209340 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.209243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" event={"ID":"179ffc7c-5199-4cc9-9a62-64678a1f4449","Type":"ContainerStarted","Data":"c683683a9309fcca234279dc1bc79ec1a925b9a5256414526cc40ae82e7dc82b"} Jul 28 12:04:28.210509 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.210478 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" event={"ID":"a90f56d7-67cd-4a18-8464-95f197f4d8c3","Type":"ContainerStarted","Data":"cbaea465aec38ba947e22584bb08e6eba5770eb96bb7e87d1c83dd4ef335d401"} Jul 28 12:04:28.211819 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.211795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" event={"ID":"ec989cab-4f7d-459a-8002-38978bd5636a","Type":"ContainerStarted","Data":"90cc01f1e5716e27bc7b37657aeecce89ce3c3cc0e58cde18525dbe638212f03"} Jul 28 12:04:28.213437 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.213414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" event={"ID":"e64e69a2-05f2-43a3-a02b-226b17f00515","Type":"ContainerStarted","Data":"fa3d2fb0ae22eb7c02db16840e6aaafdf5aa449f0435d50c1ed2bd22b309205c"} Jul 28 12:04:28.216739 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.215339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57ff94bd4-krfz9" event={"ID":"6ef4d210-7700-4a6a-8c63-f734c6dd4add","Type":"ContainerStarted","Data":"8e0a4e6e0e9f0a494e6a589fad940170b5dc17648c4dffd07e1ac3429392a2bc"} Jul 28 12:04:28.216739 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.215376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57ff94bd4-krfz9" event={"ID":"6ef4d210-7700-4a6a-8c63-f734c6dd4add","Type":"ContainerStarted","Data":"f9c3536a5ec0c2ecc3c48fbf66a568e44ba509e8236daf72d5d80f1b9521dddc"} Jul 28 12:04:28.244405 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.244360 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57ff94bd4-krfz9" podStartSLOduration=2.244346051 podStartE2EDuration="2.244346051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:28.241547841 +0000 UTC m=+82.058477660" watchObservedRunningTime="2026-07-28 12:04:28.244346051 +0000 UTC m=+82.061275867" Jul 28 12:04:28.260692 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.260668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:28.460254 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:28.460223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 28 12:04:29.222176 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:29.222145 2564 generic.go:356] "Generic (PLEG): container finished" podID="95ac02ce-43e4-4f24-9bad-f5936f09028c" containerID="70dfa3c1e530e425389764c974e463cc952de160f52a8292081c8004e00d80ed" exitCode=0 Jul 28 12:04:29.222698 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:29.222236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerDied","Data":"70dfa3c1e530e425389764c974e463cc952de160f52a8292081c8004e00d80ed"} Jul 28 12:04:29.222698 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:29.222328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerStarted","Data":"448c3f9a752848e93e240053a52b752af58f8482b4b45505ab36dfc32d969547"} Jul 28 12:04:31.233865 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.233820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" event={"ID":"e64e69a2-05f2-43a3-a02b-226b17f00515","Type":"ContainerStarted","Data":"dede53b73e3dc8dc6e48ebfca9cbedcc21995e3414cb0da49f59d2e7452ae0f1"} Jul 28 12:04:31.234298 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.233916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" event={"ID":"e64e69a2-05f2-43a3-a02b-226b17f00515","Type":"ContainerStarted","Data":"05d9cb9520c4cf86cfb229cb2286f6fdc81f1822628a9f6c45b65fac1e34177f"} Jul 28 12:04:31.234298 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.233938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" event={"ID":"e64e69a2-05f2-43a3-a02b-226b17f00515","Type":"ContainerStarted","Data":"aef3e9f61bae706ad076556b6cd7361647c812bed83cf4ebf7c6051baeb39c00"} Jul 28 12:04:31.237659 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.237625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"efd872fc-5cef-4712-b468-3ea71bcd14f2","Type":"ContainerStarted","Data":"eca4144084da6e108100f358a1af68fdbeed1e3959247d1bf71c8c0275a94d46"} Jul 28 12:04:31.241238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.241203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" event={"ID":"179ffc7c-5199-4cc9-9a62-64678a1f4449","Type":"ContainerStarted","Data":"9582614f8cbe600c53088c35685d5767803c46d5915401b5cbc4719bf8232061"} Jul 28 12:04:31.241345 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.241274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" event={"ID":"179ffc7c-5199-4cc9-9a62-64678a1f4449","Type":"ContainerStarted","Data":"330e238754b85124620520fc3886c583a672b03bc97fdc1bd99dad130f9633df"} Jul 28 12:04:31.241345 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.241328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:31.241448 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.241366 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" event={"ID":"179ffc7c-5199-4cc9-9a62-64678a1f4449","Type":"ContainerStarted","Data":"6f2357504bf047ad89546a3e71f93cd2fafb19345ff3f39fbc88d11e2bee1ba3"} Jul 28 12:04:31.243155 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.243131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" event={"ID":"a90f56d7-67cd-4a18-8464-95f197f4d8c3","Type":"ContainerStarted","Data":"8768073aa5f3ee42adac9ae6886fb47e8bf031901a1ca09d3e69c6878d676f3f"} Jul 28 12:04:31.243590 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.243547 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:31.244918 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.244868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" event={"ID":"ec989cab-4f7d-459a-8002-38978bd5636a","Type":"ContainerStarted","Data":"154055ae6641b0f6b537df419ade53e7404d34714ed8fabaab4c64b08ecb8852"} Jul 28 12:04:31.248692 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.248649 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" Jul 28 12:04:31.272474 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.272423 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b4996fbf9-dmx8d" podStartSLOduration=2.877861457 podStartE2EDuration="5.272404886s" totalImagesPullingTime="2.394543429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:31.269907686 +0000 UTC m=+85.086837503" watchObservedRunningTime="2026-07-28 12:04:31.272404886 +0000 UTC m=+85.089334704" Jul 28 12:04:31.302462 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.302409 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.25393436 podStartE2EDuration="9.302392339s" totalImagesPullingTime="6.027743722s" totalInitContainerRuntime="20.714257ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:31.300758253 +0000 UTC m=+85.117688070" watchObservedRunningTime="2026-07-28 12:04:31.302392339 +0000 UTC m=+85.119322190" Jul 28 12:04:31.325552 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.325495 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" podStartSLOduration=2.72939466 podStartE2EDuration="5.325470091s" totalImagesPullingTime="2.596075431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:31.324035793 +0000 UTC m=+85.140965611" watchObservedRunningTime="2026-07-28 12:04:31.325470091 +0000 UTC m=+85.142399911" Jul 28 12:04:31.355642 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:31.355593 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" podStartSLOduration=2.5995478910000003 podStartE2EDuration="7.355576668s" totalImagesPullingTime="4.756028777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:31.35544187 +0000 UTC m=+85.172371688" watchObservedRunningTime="2026-07-28 12:04:31.355576668 +0000 UTC m=+85.172506485" Jul 28 12:04:32.139231 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:32.139160 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-mv4k4" podStartSLOduration=3.800598583 podStartE2EDuration="6.139136935s" totalImagesPullingTime="2.338538352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:31.37604678 +0000 UTC m=+85.192976594" watchObservedRunningTime="2026-07-28 12:04:32.139136935 +0000 UTC m=+85.956066753" Jul 28 12:04:32.139593 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:32.139576 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57ff94bd4-krfz9"] Jul 28 12:04:32.259277 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:32.259253 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c556dff8f-52c5j" Jul 28 12:04:34.263887 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:34.263853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerStarted","Data":"5a2ec05f1174289ec4b2c308fedc471fd9c7fd70caa2a18568a1f9b7c2254268"} Jul 28 12:04:34.264265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:34.263958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerStarted","Data":"9ee5d2f2b02b5c10c000cc04ff623c60c6bfa7d2f95ee647fc13c6a598956ab3"} Jul 28 12:04:34.264265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:34.263986 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerStarted","Data":"bc9917651163063db7e823df3f132dddb756f7312be7627136f5b513f422c393"} Jul 28 12:04:34.264265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:34.264009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerStarted","Data":"7e073176fafe08f0d99f8ef8daf41f05f426f322f7271da4b39d65d423a51eb5"} Jul 28 12:04:34.264265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:34.264033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerStarted","Data":"69b7cab48712317b7cb670932955b511a04b77a42624d49c57601ef42bf4001b"} Jul 28 12:04:34.264265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:34.264057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95ac02ce-43e4-4f24-9bad-f5936f09028c","Type":"ContainerStarted","Data":"f22d60a94ffe940db134b9c8a6ef6421b0c30b8ded9d6107ba5d3cd7ec938c50"} Jul 28 12:04:34.297020 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:34.296971 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.204378217 podStartE2EDuration="7.296955909s" totalImagesPullingTime="4.058877095s" totalInitContainerRuntime="33.700597ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:34.294931176 +0000 UTC m=+88.111860995" watchObservedRunningTime="2026-07-28 12:04:34.296955909 +0000 UTC m=+88.113885727" Jul 28 12:04:36.499116 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:36.499071 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:37.232007 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:37.231968 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vqt77_bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42/serve-healthcheck-canary/0.log" Jul 28 12:04:38.261842 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:38.261804 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:04:43.499577 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:43.499541 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff44d47d7-9vsnq"] Jul 28 12:04:44.240339 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:44.240295 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-df698df75-7sp9r" podUID="9abbcf32-0d25-4dca-88df-1fab1334fdcd" containerName="console" containerID="cri-o://8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc" gracePeriod=15 Jul 28 12:04:45.047308 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.047276 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:45.102895 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.102852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-service-ca\") pod \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " Jul 28 12:04:45.103097 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.102982 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-oauth-config\") pod \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " Jul 28 12:04:45.103097 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.103014 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs2tw\" (UniqueName: \"kubernetes.io/projected/9abbcf32-0d25-4dca-88df-1fab1334fdcd-kube-api-access-rs2tw\") pod \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " Jul 28 12:04:45.103097 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.103044 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-oauth-serving-cert\") pod \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " Jul 28 12:04:45.103097 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.103068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-config\") pod \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " Jul 28 12:04:45.103304 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.103150 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-serving-cert\") pod \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\" (UID: \"9abbcf32-0d25-4dca-88df-1fab1334fdcd\") " Jul 28 12:04:45.103452 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.103411 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9abbcf32-0d25-4dca-88df-1fab1334fdcd" (UID: "9abbcf32-0d25-4dca-88df-1fab1334fdcd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:04:45.103549 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.103420 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-service-ca" (OuterVolumeSpecName: "service-ca") pod "9abbcf32-0d25-4dca-88df-1fab1334fdcd" (UID: "9abbcf32-0d25-4dca-88df-1fab1334fdcd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:04:45.103549 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.103483 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-config" (OuterVolumeSpecName: "console-config") pod "9abbcf32-0d25-4dca-88df-1fab1334fdcd" (UID: "9abbcf32-0d25-4dca-88df-1fab1334fdcd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:04:45.105707 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.105662 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9abbcf32-0d25-4dca-88df-1fab1334fdcd" (UID: "9abbcf32-0d25-4dca-88df-1fab1334fdcd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:04:45.105707 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.105688 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abbcf32-0d25-4dca-88df-1fab1334fdcd-kube-api-access-rs2tw" (OuterVolumeSpecName: "kube-api-access-rs2tw") pod "9abbcf32-0d25-4dca-88df-1fab1334fdcd" (UID: "9abbcf32-0d25-4dca-88df-1fab1334fdcd"). InnerVolumeSpecName "kube-api-access-rs2tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:04:45.105858 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.105726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9abbcf32-0d25-4dca-88df-1fab1334fdcd" (UID: "9abbcf32-0d25-4dca-88df-1fab1334fdcd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:04:45.204320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.204229 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-oauth-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:45.204320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.204259 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs2tw\" (UniqueName: \"kubernetes.io/projected/9abbcf32-0d25-4dca-88df-1fab1334fdcd-kube-api-access-rs2tw\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:45.204320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.204270 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-oauth-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:45.204320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.204278 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:45.204320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.204289 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9abbcf32-0d25-4dca-88df-1fab1334fdcd-console-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:45.204320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.204296 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9abbcf32-0d25-4dca-88df-1fab1334fdcd-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:45.317657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.317619 2564 generic.go:356] "Generic (PLEG): container finished" podID="9abbcf32-0d25-4dca-88df-1fab1334fdcd" containerID="8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc" exitCode=0 Jul 28 12:04:45.317831 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.317682 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df698df75-7sp9r" Jul 28 12:04:45.317831 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.317696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df698df75-7sp9r" event={"ID":"9abbcf32-0d25-4dca-88df-1fab1334fdcd","Type":"ContainerDied","Data":"8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc"} Jul 28 12:04:45.317831 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.317776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df698df75-7sp9r" event={"ID":"9abbcf32-0d25-4dca-88df-1fab1334fdcd","Type":"ContainerDied","Data":"f64d4bab7c2b62655fb9040bcb43e2b52886e2fbae1c11859e47c11e105f7f72"} Jul 28 12:04:45.317831 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.317797 2564 scope.go:117] "RemoveContainer" containerID="8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc" Jul 28 12:04:45.326618 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.326604 2564 scope.go:117] "RemoveContainer" containerID="8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc" Jul 28 12:04:45.326858 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:04:45.326842 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc\": container with ID starting with 8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc not found: ID does not exist" containerID="8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc" Jul 28 12:04:45.326919 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.326866 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc"} err="failed to get container status \"8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc\": rpc error: code = NotFound desc = could not find container \"8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc\": container with ID starting with 8a699477b359826db6feadac2745e943bb325c0d8cbd4b05ad77b087672c2edc not found: ID does not exist" Jul 28 12:04:45.340741 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.340715 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df698df75-7sp9r"] Jul 28 12:04:45.343760 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:45.343740 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-df698df75-7sp9r"] Jul 28 12:04:46.601238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:46.601206 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:46.601632 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:46.601265 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:04:46.780937 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:46.780903 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abbcf32-0d25-4dca-88df-1fab1334fdcd" path="/var/lib/kubelet/pods/9abbcf32-0d25-4dca-88df-1fab1334fdcd/volumes" Jul 28 12:04:49.135000 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:49.134970 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hmlhv" Jul 28 12:04:57.160674 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.160631 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57ff94bd4-krfz9" podUID="6ef4d210-7700-4a6a-8c63-f734c6dd4add" containerName="console" containerID="cri-o://8e0a4e6e0e9f0a494e6a589fad940170b5dc17648c4dffd07e1ac3429392a2bc" gracePeriod=15 Jul 28 12:04:57.385052 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.385020 2564 generic.go:356] "Generic (PLEG): container finished" podID="6ef4d210-7700-4a6a-8c63-f734c6dd4add" containerID="8e0a4e6e0e9f0a494e6a589fad940170b5dc17648c4dffd07e1ac3429392a2bc" exitCode=0 Jul 28 12:04:57.385180 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.385058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57ff94bd4-krfz9" event={"ID":"6ef4d210-7700-4a6a-8c63-f734c6dd4add","Type":"ContainerDied","Data":"8e0a4e6e0e9f0a494e6a589fad940170b5dc17648c4dffd07e1ac3429392a2bc"} Jul 28 12:04:57.437393 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.437331 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:57.506151 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506129 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-oauth-serving-cert\") pod \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " Jul 28 12:04:57.506262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506174 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-serving-cert\") pod \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " Jul 28 12:04:57.506262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-service-ca\") pod \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " Jul 28 12:04:57.506262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506247 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbqz2\" (UniqueName: \"kubernetes.io/projected/6ef4d210-7700-4a6a-8c63-f734c6dd4add-kube-api-access-hbqz2\") pod \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " Jul 28 12:04:57.506390 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506289 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-oauth-config\") pod \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " Jul 28 12:04:57.506390 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506313 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-trusted-ca-bundle\") pod \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " Jul 28 12:04:57.506390 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506384 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-config\") pod \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\" (UID: \"6ef4d210-7700-4a6a-8c63-f734c6dd4add\") " Jul 28 12:04:57.506641 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506618 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6ef4d210-7700-4a6a-8c63-f734c6dd4add" (UID: "6ef4d210-7700-4a6a-8c63-f734c6dd4add"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:04:57.506726 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506656 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-service-ca" (OuterVolumeSpecName: "service-ca") pod "6ef4d210-7700-4a6a-8c63-f734c6dd4add" (UID: "6ef4d210-7700-4a6a-8c63-f734c6dd4add"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:04:57.506801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506775 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6ef4d210-7700-4a6a-8c63-f734c6dd4add" (UID: "6ef4d210-7700-4a6a-8c63-f734c6dd4add"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:04:57.506936 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.506872 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-config" (OuterVolumeSpecName: "console-config") pod "6ef4d210-7700-4a6a-8c63-f734c6dd4add" (UID: "6ef4d210-7700-4a6a-8c63-f734c6dd4add"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:04:57.508434 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.508404 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6ef4d210-7700-4a6a-8c63-f734c6dd4add" (UID: "6ef4d210-7700-4a6a-8c63-f734c6dd4add"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:04:57.508521 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.508496 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef4d210-7700-4a6a-8c63-f734c6dd4add-kube-api-access-hbqz2" (OuterVolumeSpecName: "kube-api-access-hbqz2") pod "6ef4d210-7700-4a6a-8c63-f734c6dd4add" (UID: "6ef4d210-7700-4a6a-8c63-f734c6dd4add"). InnerVolumeSpecName "kube-api-access-hbqz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:04:57.508656 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.508639 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6ef4d210-7700-4a6a-8c63-f734c6dd4add" (UID: "6ef4d210-7700-4a6a-8c63-f734c6dd4add"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:04:57.607127 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.607107 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-oauth-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:57.607127 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.607126 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:57.607238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.607136 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:57.607238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.607144 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbqz2\" (UniqueName: \"kubernetes.io/projected/6ef4d210-7700-4a6a-8c63-f734c6dd4add-kube-api-access-hbqz2\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:57.607238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.607152 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-oauth-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:57.607238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.607161 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-trusted-ca-bundle\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:57.607238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:57.607169 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6ef4d210-7700-4a6a-8c63-f734c6dd4add-console-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:04:58.394221 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:58.394186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57ff94bd4-krfz9" event={"ID":"6ef4d210-7700-4a6a-8c63-f734c6dd4add","Type":"ContainerDied","Data":"f9c3536a5ec0c2ecc3c48fbf66a568e44ba509e8236daf72d5d80f1b9521dddc"} Jul 28 12:04:58.394657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:58.394264 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57ff94bd4-krfz9" Jul 28 12:04:58.394657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:58.394272 2564 scope.go:117] "RemoveContainer" containerID="8e0a4e6e0e9f0a494e6a589fad940170b5dc17648c4dffd07e1ac3429392a2bc" Jul 28 12:04:58.435352 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:58.435327 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57ff94bd4-krfz9"] Jul 28 12:04:58.441614 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:58.441592 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57ff94bd4-krfz9"] Jul 28 12:04:58.779831 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:04:58.779796 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef4d210-7700-4a6a-8c63-f734c6dd4add" path="/var/lib/kubelet/pods/6ef4d210-7700-4a6a-8c63-f734c6dd4add/volumes" Jul 28 12:05:06.606378 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:06.606350 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:05:06.610273 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:06.610250 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-768c457df4-csh9b" Jul 28 12:05:08.524363 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.524320 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6ff44d47d7-9vsnq" podUID="84d5cb74-a790-4feb-bcd1-19df7a193ad0" containerName="console" containerID="cri-o://bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712" gracePeriod=15 Jul 28 12:05:08.775532 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.775480 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:05:08.897827 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.897801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-service-ca\") pod \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " Jul 28 12:05:08.897972 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.897856 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-trusted-ca-bundle\") pod \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " Jul 28 12:05:08.897972 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.897912 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-oauth-config\") pod \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " Jul 28 12:05:08.897972 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.897951 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-config\") pod \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " Jul 28 12:05:08.898141 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898126 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbc6m\" (UniqueName: \"kubernetes.io/projected/84d5cb74-a790-4feb-bcd1-19df7a193ad0-kube-api-access-fbc6m\") pod \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " Jul 28 12:05:08.898195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898179 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-serving-cert\") pod \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " Jul 28 12:05:08.898252 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-oauth-serving-cert\") pod \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\" (UID: \"84d5cb74-a790-4feb-bcd1-19df7a193ad0\") " Jul 28 12:05:08.898305 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-service-ca" (OuterVolumeSpecName: "service-ca") pod "84d5cb74-a790-4feb-bcd1-19df7a193ad0" (UID: "84d5cb74-a790-4feb-bcd1-19df7a193ad0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:05:08.898361 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898301 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-config" (OuterVolumeSpecName: "console-config") pod "84d5cb74-a790-4feb-bcd1-19df7a193ad0" (UID: "84d5cb74-a790-4feb-bcd1-19df7a193ad0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:05:08.898361 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898345 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "84d5cb74-a790-4feb-bcd1-19df7a193ad0" (UID: "84d5cb74-a790-4feb-bcd1-19df7a193ad0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:05:08.898557 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898532 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:05:08.898629 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898562 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-trusted-ca-bundle\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:05:08.898629 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898577 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:05:08.898629 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.898603 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "84d5cb74-a790-4feb-bcd1-19df7a193ad0" (UID: "84d5cb74-a790-4feb-bcd1-19df7a193ad0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:05:08.900282 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.900259 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d5cb74-a790-4feb-bcd1-19df7a193ad0-kube-api-access-fbc6m" (OuterVolumeSpecName: "kube-api-access-fbc6m") pod "84d5cb74-a790-4feb-bcd1-19df7a193ad0" (UID: "84d5cb74-a790-4feb-bcd1-19df7a193ad0"). InnerVolumeSpecName "kube-api-access-fbc6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:05:08.900822 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.900791 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "84d5cb74-a790-4feb-bcd1-19df7a193ad0" (UID: "84d5cb74-a790-4feb-bcd1-19df7a193ad0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:05:08.900910 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.900837 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "84d5cb74-a790-4feb-bcd1-19df7a193ad0" (UID: "84d5cb74-a790-4feb-bcd1-19df7a193ad0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:05:08.998913 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.998872 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:05:08.999010 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.998915 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84d5cb74-a790-4feb-bcd1-19df7a193ad0-oauth-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:05:08.999010 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.998928 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84d5cb74-a790-4feb-bcd1-19df7a193ad0-console-oauth-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:05:08.999010 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:08.998940 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbc6m\" (UniqueName: \"kubernetes.io/projected/84d5cb74-a790-4feb-bcd1-19df7a193ad0-kube-api-access-fbc6m\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:05:09.446201 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.446169 2564 generic.go:356] "Generic (PLEG): container finished" podID="84d5cb74-a790-4feb-bcd1-19df7a193ad0" containerID="bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712" exitCode=0 Jul 28 12:05:09.446347 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.446210 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff44d47d7-9vsnq" Jul 28 12:05:09.446347 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.446232 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff44d47d7-9vsnq" event={"ID":"84d5cb74-a790-4feb-bcd1-19df7a193ad0","Type":"ContainerDied","Data":"bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712"} Jul 28 12:05:09.446347 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.446278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff44d47d7-9vsnq" event={"ID":"84d5cb74-a790-4feb-bcd1-19df7a193ad0","Type":"ContainerDied","Data":"aead7d94180b344493a5247f3c483ded8d3db898de0ea5aa0d48ce8894ac5fef"} Jul 28 12:05:09.446347 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.446300 2564 scope.go:117] "RemoveContainer" containerID="bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712" Jul 28 12:05:09.455431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.455412 2564 scope.go:117] "RemoveContainer" containerID="bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712" Jul 28 12:05:09.455673 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:05:09.455653 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712\": container with ID starting with bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712 not found: ID does not exist" containerID="bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712" Jul 28 12:05:09.455723 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.455680 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712"} err="failed to get container status \"bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712\": rpc error: code = NotFound desc = could not find container \"bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712\": container with ID starting with bddef80d6d021a3962e7768a7f0308038c5069c60e9a86e212669fb641e63712 not found: ID does not exist" Jul 28 12:05:09.474529 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.474507 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff44d47d7-9vsnq"] Jul 28 12:05:09.487466 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:09.487448 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ff44d47d7-9vsnq"] Jul 28 12:05:10.779960 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:10.779915 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d5cb74-a790-4feb-bcd1-19df7a193ad0" path="/var/lib/kubelet/pods/84d5cb74-a790-4feb-bcd1-19df7a193ad0/volumes" Jul 28 12:05:28.261928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:28.261871 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:05:28.281727 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:28.281703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:05:28.540817 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:05:28.540748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 28 12:08:06.589346 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:08:06.589305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:08:06.593418 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:08:06.593397 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:08:06.596501 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:08:06.596475 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 12:09:43.349077 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349049 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl"] Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349389 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abbcf32-0d25-4dca-88df-1fab1334fdcd" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349400 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abbcf32-0d25-4dca-88df-1fab1334fdcd" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349423 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d5cb74-a790-4feb-bcd1-19df7a193ad0" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349429 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d5cb74-a790-4feb-bcd1-19df7a193ad0" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349442 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef4d210-7700-4a6a-8c63-f734c6dd4add" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349448 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef4d210-7700-4a6a-8c63-f734c6dd4add" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349498 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d5cb74-a790-4feb-bcd1-19df7a193ad0" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349505 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef4d210-7700-4a6a-8c63-f734c6dd4add" containerName="console" Jul 28 12:09:43.349514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.349511 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abbcf32-0d25-4dca-88df-1fab1334fdcd" containerName="console" Jul 28 12:09:43.352000 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.351985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.357109 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.357080 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 28 12:09:43.357526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.357496 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 12:09:43.357630 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.357562 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 12:09:43.357737 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.357720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jm2sz\"" Jul 28 12:09:43.357804 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.357792 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 28 12:09:43.358110 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.358086 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 28 12:09:43.358553 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.358250 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 28 12:09:43.358553 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.358316 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 28 12:09:43.369235 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.369214 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl"] Jul 28 12:09:43.438094 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.438068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxcw\" (UniqueName: \"kubernetes.io/projected/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-kube-api-access-vlxcw\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.438199 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.438102 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.438199 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.438121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.438199 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.438135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-online-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.438199 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.438167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-feast-data\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.438199 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.438198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.538860 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.538836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.538983 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.538938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxcw\" (UniqueName: \"kubernetes.io/projected/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-kube-api-access-vlxcw\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.539047 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.538982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.539047 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:43.538988 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 28 12:09:43.539047 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.539013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.539047 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.539038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-online-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.539238 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:43.539070 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls podName:824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd nodeName:}" failed. No retries permitted until 2026-07-28 12:09:44.039045496 +0000 UTC m=+397.855975292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls") pod "feast-simple-feast-setup-5d594c7856-zncrl" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd") : secret "feast-simple-feast-setup-ui-tls" not found Jul 28 12:09:43.539238 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.539119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-feast-data\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.539238 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:43.539132 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 28 12:09:43.539238 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:43.539189 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls podName:824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd nodeName:}" failed. No retries permitted until 2026-07-28 12:09:44.039172411 +0000 UTC m=+397.856102206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls") pod "feast-simple-feast-setup-5d594c7856-zncrl" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd") : secret "feast-simple-feast-setup-registry-tls" not found Jul 28 12:09:43.539436 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:43.539243 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 28 12:09:43.539436 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:43.539299 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls podName:824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd nodeName:}" failed. No retries permitted until 2026-07-28 12:09:44.039289355 +0000 UTC m=+397.856219151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls") pod "feast-simple-feast-setup-5d594c7856-zncrl" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd") : secret "feast-simple-feast-setup-offline-tls" not found Jul 28 12:09:43.539523 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.539502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-feast-data\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.541481 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.541461 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-online-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:43.548357 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:43.548338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxcw\" (UniqueName: \"kubernetes.io/projected/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-kube-api-access-vlxcw\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:44.043355 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:44.043313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:44.043536 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:44.043417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:44.043536 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:44.043449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:44.043663 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:44.043552 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 28 12:09:44.043663 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:44.043582 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 28 12:09:44.043663 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:44.043629 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls podName:824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd nodeName:}" failed. No retries permitted until 2026-07-28 12:09:45.043609993 +0000 UTC m=+398.860539801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls") pod "feast-simple-feast-setup-5d594c7856-zncrl" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd") : secret "feast-simple-feast-setup-registry-tls" not found Jul 28 12:09:44.043663 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:09:44.043656 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls podName:824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd nodeName:}" failed. No retries permitted until 2026-07-28 12:09:45.043637003 +0000 UTC m=+398.860566815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls") pod "feast-simple-feast-setup-5d594c7856-zncrl" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd") : secret "feast-simple-feast-setup-offline-tls" not found Jul 28 12:09:44.045888 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:44.045859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:45.051799 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.051769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:45.051799 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.051804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:45.054182 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.054162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:45.054267 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.054221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-zncrl\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:45.171017 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.170989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:09:45.308956 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.308935 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl"] Jul 28 12:09:45.311005 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:09:45.310978 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod824aa5ab_52fa_4fb1_aa37_d4a38ee05fdd.slice/crio-2f69294aa04a647c2c3200cd2f0e42ec9b600b92255e286e39a0456188693bd8 WatchSource:0}: Error finding container 2f69294aa04a647c2c3200cd2f0e42ec9b600b92255e286e39a0456188693bd8: Status 404 returned error can't find the container with id 2f69294aa04a647c2c3200cd2f0e42ec9b600b92255e286e39a0456188693bd8 Jul 28 12:09:45.312932 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.312912 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:09:45.563944 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:09:45.563865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerStarted","Data":"2f69294aa04a647c2c3200cd2f0e42ec9b600b92255e286e39a0456188693bd8"} Jul 28 12:10:14.693072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:14.692274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerStarted","Data":"ecda57307bee858daeaba3d0c6be2b42511f5e3f8afcf3d513a345e2db1b3837"} Jul 28 12:10:19.712385 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:19.712351 2564 generic.go:356] "Generic (PLEG): container finished" podID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerID="ecda57307bee858daeaba3d0c6be2b42511f5e3f8afcf3d513a345e2db1b3837" exitCode=0 Jul 28 12:10:19.712803 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:19.712415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerDied","Data":"ecda57307bee858daeaba3d0c6be2b42511f5e3f8afcf3d513a345e2db1b3837"} Jul 28 12:10:20.718453 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:20.718412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerStarted","Data":"3cd63121a095c81dd0da943014af76c992747bb3e0e2918f8e38cab3c7c96a94"} Jul 28 12:10:25.739435 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:25.739402 2564 generic.go:356] "Generic (PLEG): container finished" podID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerID="3cd63121a095c81dd0da943014af76c992747bb3e0e2918f8e38cab3c7c96a94" exitCode=0 Jul 28 12:10:25.739787 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:25.739477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerDied","Data":"3cd63121a095c81dd0da943014af76c992747bb3e0e2918f8e38cab3c7c96a94"} Jul 28 12:10:25.750921 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:25.750902 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 28 12:10:26.749816 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:26.749777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerStarted","Data":"b0a84fe3c494105b48ac081082e3c4b283f76224e14643fc2fbd1fefe4cf4ef4"} Jul 28 12:10:26.750301 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:26.749836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerStarted","Data":"b5f28b5adc360b386a1d3933ba0fdea6c16617ef8fb1429696992143ecbd82ba"} Jul 28 12:10:26.750301 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:26.749857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerStarted","Data":"527b317d3d422201a7173c1e0439fbf44930b78fa17921dd0fbb898fe6a62cb7"} Jul 28 12:10:26.750301 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:26.749903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerStarted","Data":"28d068fa9b4a6d5b8627eca3aac6431ffb986f590a72ecd0386570580072691e"} Jul 28 12:10:26.787771 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:26.787715 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podStartSLOduration=14.684739288 podStartE2EDuration="43.787693953s" totalImagesPullingTime="29.102954665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:10:26.786742219 +0000 UTC m=+440.603672049" watchObservedRunningTime="2026-07-28 12:10:26.787693953 +0000 UTC m=+440.604623770" Jul 28 12:10:27.171801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.171548 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:27.171801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.171612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:27.171801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.171642 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:27.171801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.171663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:27.173761 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.173697 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jul 28 12:10:27.173761 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.173726 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused" Jul 28 12:10:27.173761 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.173749 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jul 28 12:10:27.174050 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:27.173847 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Jul 28 12:10:30.171818 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:30.171747 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Jul 28 12:10:30.171818 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:30.171768 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jul 28 12:10:30.171818 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:30.171784 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jul 28 12:10:30.171818 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:30.171768 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused" Jul 28 12:10:33.172094 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.172061 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.172633 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.172252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.172689 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.172636 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.172689 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.172672 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.172782 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.172690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.172782 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.172705 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.173084 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.173058 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.173199 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.173140 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.173237 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.173211 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.177374 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.177359 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.786897 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.786853 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:10:33.790056 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:10:33.790033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:11:36.874192 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:36.874157 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl"] Jul 28 12:11:36.874697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:36.874464 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="registry" containerID="cri-o://28d068fa9b4a6d5b8627eca3aac6431ffb986f590a72ecd0386570580072691e" gracePeriod=30 Jul 28 12:11:36.874697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:36.874481 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" containerID="cri-o://b0a84fe3c494105b48ac081082e3c4b283f76224e14643fc2fbd1fefe4cf4ef4" gracePeriod=30 Jul 28 12:11:36.874697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:36.874481 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" containerID="cri-o://527b317d3d422201a7173c1e0439fbf44930b78fa17921dd0fbb898fe6a62cb7" gracePeriod=30 Jul 28 12:11:36.874697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:36.874481 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="offline" containerID="cri-o://b5f28b5adc360b386a1d3933ba0fdea6c16617ef8fb1429696992143ecbd82ba" gracePeriod=30 Jul 28 12:11:37.264070 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.264037 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf"] Jul 28 12:11:37.267332 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.267311 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.270520 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.270495 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-2jp64\"" Jul 28 12:11:37.284848 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.284818 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf"] Jul 28 12:11:37.381940 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.381908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.382078 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.381980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/75dc0468-0375-4514-9821-8b42e225a3d5-feast-data\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.382078 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.382007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-ui-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.382078 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.382051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhkmm\" (UniqueName: \"kubernetes.io/projected/75dc0468-0375-4514-9821-8b42e225a3d5-kube-api-access-mhkmm\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.382214 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.382115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-online-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.382214 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.382148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.483174 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.483142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-online-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.483174 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.483176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.483387 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.483312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.483387 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.483378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/75dc0468-0375-4514-9821-8b42e225a3d5-feast-data\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.483495 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.483408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-ui-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.483495 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.483460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhkmm\" (UniqueName: \"kubernetes.io/projected/75dc0468-0375-4514-9821-8b42e225a3d5-kube-api-access-mhkmm\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.483781 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.483752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/75dc0468-0375-4514-9821-8b42e225a3d5-feast-data\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.485867 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.485835 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-registry-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.486000 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.485943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-ui-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.486051 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.486003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-offline-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.486146 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.486127 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-online-tls\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.492775 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.492753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhkmm\" (UniqueName: \"kubernetes.io/projected/75dc0468-0375-4514-9821-8b42e225a3d5-kube-api-access-mhkmm\") pod \"feast-simple-feast-setup-5d594c7856-jhzxf\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.578621 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.578587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:37.718095 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:37.718062 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf"] Jul 28 12:11:37.720166 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:11:37.720141 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75dc0468_0375_4514_9821_8b42e225a3d5.slice/crio-95d95c28131e586e5294884c7ddf91a3e2cb575338e035716c7123eafa6532b9 WatchSource:0}: Error finding container 95d95c28131e586e5294884c7ddf91a3e2cb575338e035716c7123eafa6532b9: Status 404 returned error can't find the container with id 95d95c28131e586e5294884c7ddf91a3e2cb575338e035716c7123eafa6532b9 Jul 28 12:11:38.050053 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:38.049962 2564 generic.go:356] "Generic (PLEG): container finished" podID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerID="b0a84fe3c494105b48ac081082e3c4b283f76224e14643fc2fbd1fefe4cf4ef4" exitCode=0 Jul 28 12:11:38.050053 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:38.049988 2564 generic.go:356] "Generic (PLEG): container finished" podID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerID="527b317d3d422201a7173c1e0439fbf44930b78fa17921dd0fbb898fe6a62cb7" exitCode=0 Jul 28 12:11:38.050053 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:38.049992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerDied","Data":"b0a84fe3c494105b48ac081082e3c4b283f76224e14643fc2fbd1fefe4cf4ef4"} Jul 28 12:11:38.050620 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:38.050064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerDied","Data":"527b317d3d422201a7173c1e0439fbf44930b78fa17921dd0fbb898fe6a62cb7"} Jul 28 12:11:38.052599 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:38.052560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerStarted","Data":"3f74b314a6d4c5b07a8c378ce08f4b6f90c0907d1889f595c23133f9b5d212b9"} Jul 28 12:11:38.052747 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:38.052620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerStarted","Data":"95d95c28131e586e5294884c7ddf91a3e2cb575338e035716c7123eafa6532b9"} Jul 28 12:11:43.075737 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:43.075707 2564 generic.go:356] "Generic (PLEG): container finished" podID="75dc0468-0375-4514-9821-8b42e225a3d5" containerID="3f74b314a6d4c5b07a8c378ce08f4b6f90c0907d1889f595c23133f9b5d212b9" exitCode=0 Jul 28 12:11:43.076245 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:43.075783 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerDied","Data":"3f74b314a6d4c5b07a8c378ce08f4b6f90c0907d1889f595c23133f9b5d212b9"} Jul 28 12:11:43.173053 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:43.173018 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jul 28 12:11:43.788003 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:43.787960 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jul 28 12:11:44.082031 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:44.081946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerStarted","Data":"43cf4ea89491cc5948ce5d29156885cf2584c957276fdd0de39d5c51613fab9a"} Jul 28 12:11:49.107523 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:49.107491 2564 generic.go:356] "Generic (PLEG): container finished" podID="75dc0468-0375-4514-9821-8b42e225a3d5" containerID="43cf4ea89491cc5948ce5d29156885cf2584c957276fdd0de39d5c51613fab9a" exitCode=0 Jul 28 12:11:49.107872 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:49.107566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerDied","Data":"43cf4ea89491cc5948ce5d29156885cf2584c957276fdd0de39d5c51613fab9a"} Jul 28 12:11:50.118180 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:50.118141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerStarted","Data":"fe25d239ce32d010cbce7b360affb661d1ddd753d43ccdcdff3f34a652e8e29a"} Jul 28 12:11:50.118657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:50.118205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerStarted","Data":"33290375be495a65f1fcbb43aaaf273e9cf45b5e1453bc66b68ec43e06b16d1d"} Jul 28 12:11:50.118657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:50.118226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerStarted","Data":"3e2257549a97d93ab13467471c08bed13336eee83da5f1e66529e6572635609b"} Jul 28 12:11:50.118657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:50.118244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerStarted","Data":"3745a06b17628b5490e6a52cc4a98bbee47f4a7eb8e677b11f03f82c45ca1277"} Jul 28 12:11:50.146693 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:50.146634 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podStartSLOduration=13.146614242 podStartE2EDuration="13.146614242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:11:50.144439058 +0000 UTC m=+523.961368876" watchObservedRunningTime="2026-07-28 12:11:50.146614242 +0000 UTC m=+523.963544066" Jul 28 12:11:52.578986 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.578944 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:52.579427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.579007 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:52.579427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.579115 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:52.579427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.579155 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:52.581018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.580979 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Jul 28 12:11:52.581137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.581021 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Jul 28 12:11:52.581137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.581048 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 28 12:11:52.581316 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:52.581141 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Jul 28 12:11:53.173271 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:53.173232 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jul 28 12:11:53.787655 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:53.787603 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jul 28 12:11:55.579824 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:55.579788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:55.580396 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:55.580371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:55.580577 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:55.580561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:55.580678 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:55.580587 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:55.580734 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:55.580689 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:55.585165 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:55.585148 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:56.152800 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:56.152772 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:56.153018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:56.152815 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:56.153018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:56.152833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:56.153154 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:56.153039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:56.153339 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:56.153311 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:11:56.155793 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:11:56.155780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:12:03.173372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:03.173278 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jul 28 12:12:03.173764 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:03.173418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:12:03.787997 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:03.787959 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jul 28 12:12:03.788159 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:03.788090 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:12:07.198843 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.198767 2564 generic.go:356] "Generic (PLEG): container finished" podID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerID="b5f28b5adc360b386a1d3933ba0fdea6c16617ef8fb1429696992143ecbd82ba" exitCode=137 Jul 28 12:12:07.198843 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.198792 2564 generic.go:356] "Generic (PLEG): container finished" podID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerID="28d068fa9b4a6d5b8627eca3aac6431ffb986f590a72ecd0386570580072691e" exitCode=137 Jul 28 12:12:07.199227 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.198843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerDied","Data":"b5f28b5adc360b386a1d3933ba0fdea6c16617ef8fb1429696992143ecbd82ba"} Jul 28 12:12:07.199227 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.198904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerDied","Data":"28d068fa9b4a6d5b8627eca3aac6431ffb986f590a72ecd0386570580072691e"} Jul 28 12:12:07.523262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.523242 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:12:07.536269 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.536251 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls\") pod \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " Jul 28 12:12:07.536360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.536283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-online-tls\") pod \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " Jul 28 12:12:07.536360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.536301 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls\") pod \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " Jul 28 12:12:07.536427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.536360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlxcw\" (UniqueName: \"kubernetes.io/projected/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-kube-api-access-vlxcw\") pod \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " Jul 28 12:12:07.536427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.536382 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls\") pod \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " Jul 28 12:12:07.536511 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.536427 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-feast-data\") pod \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\" (UID: \"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd\") " Jul 28 12:12:07.537275 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.537226 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-feast-data" (OuterVolumeSpecName: "feast-data") pod "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:12:07.538588 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.538564 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-online-tls" (OuterVolumeSpecName: "online-tls") pod "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:12:07.538693 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.538601 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:12:07.538693 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.538676 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-kube-api-access-vlxcw" (OuterVolumeSpecName: "kube-api-access-vlxcw") pod "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd"). InnerVolumeSpecName "kube-api-access-vlxcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:12:07.538980 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.538953 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:12:07.539150 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.539131 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" (UID: "824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:12:07.637725 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.637696 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlxcw\" (UniqueName: \"kubernetes.io/projected/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-kube-api-access-vlxcw\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:12:07.637725 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.637722 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:12:07.637903 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.637737 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:12:07.637903 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.637749 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:12:07.637903 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.637760 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:12:07.637903 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:07.637771 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:12:08.205485 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.205450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" event={"ID":"824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd","Type":"ContainerDied","Data":"2f69294aa04a647c2c3200cd2f0e42ec9b600b92255e286e39a0456188693bd8"} Jul 28 12:12:08.205485 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.205476 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl" Jul 28 12:12:08.205976 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.205511 2564 scope.go:117] "RemoveContainer" containerID="b0a84fe3c494105b48ac081082e3c4b283f76224e14643fc2fbd1fefe4cf4ef4" Jul 28 12:12:08.214491 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.214476 2564 scope.go:117] "RemoveContainer" containerID="b5f28b5adc360b386a1d3933ba0fdea6c16617ef8fb1429696992143ecbd82ba" Jul 28 12:12:08.222054 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.222038 2564 scope.go:117] "RemoveContainer" containerID="527b317d3d422201a7173c1e0439fbf44930b78fa17921dd0fbb898fe6a62cb7" Jul 28 12:12:08.228742 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.228721 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl"] Jul 28 12:12:08.229391 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.229378 2564 scope.go:117] "RemoveContainer" containerID="28d068fa9b4a6d5b8627eca3aac6431ffb986f590a72ecd0386570580072691e" Jul 28 12:12:08.233542 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.233524 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-zncrl"] Jul 28 12:12:08.236950 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.236936 2564 scope.go:117] "RemoveContainer" containerID="3cd63121a095c81dd0da943014af76c992747bb3e0e2918f8e38cab3c7c96a94" Jul 28 12:12:08.243801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.243787 2564 scope.go:117] "RemoveContainer" containerID="ecda57307bee858daeaba3d0c6be2b42511f5e3f8afcf3d513a345e2db1b3837" Jul 28 12:12:08.781150 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:08.781120 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" path="/var/lib/kubelet/pods/824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd/volumes" Jul 28 12:12:59.061837 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.061805 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm"] Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062143 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="feast-apply" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062154 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="feast-apply" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062174 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="feast-init" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062180 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="feast-init" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062188 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062194 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062206 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="offline" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062211 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="offline" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062217 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="registry" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062223 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="registry" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062234 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062239 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062286 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="online" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062294 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="ui" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062300 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="offline" Jul 28 12:12:59.062327 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.062306 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="824aa5ab-52fa-4fb1-aa37-d4a38ee05fdd" containerName="registry" Jul 28 12:12:59.065358 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.065343 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.068208 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.068186 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 28 12:12:59.068339 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.068302 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 28 12:12:59.068389 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.068346 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 28 12:12:59.068438 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.068409 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 28 12:12:59.068584 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.068567 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-h2xfm\"" Jul 28 12:12:59.068676 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.068659 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 28 12:12:59.068797 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.068785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 28 12:12:59.069084 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.069072 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 28 12:12:59.081603 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.081576 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm"] Jul 28 12:12:59.233746 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.233716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9c532a4a-1695-4db8-93f1-b84438c2e468-feast-data\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.233868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.233751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.233868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.233783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-ui-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.233868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.233842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9c532a4a-1695-4db8-93f1-b84438c2e468-registry-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.234012 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.233896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.234012 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.233977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9lnj\" (UniqueName: \"kubernetes.io/projected/9c532a4a-1695-4db8-93f1-b84438c2e468-kube-api-access-c9lnj\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335024 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.334963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335024 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.334998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-ui-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335024 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.335021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9c532a4a-1695-4db8-93f1-b84438c2e468-registry-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335202 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.335046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335202 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.335110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9lnj\" (UniqueName: \"kubernetes.io/projected/9c532a4a-1695-4db8-93f1-b84438c2e468-kube-api-access-c9lnj\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335202 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:12:59.335126 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 28 12:12:59.335202 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.335146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9c532a4a-1695-4db8-93f1-b84438c2e468-feast-data\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335355 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:12:59.335221 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls podName:9c532a4a-1695-4db8-93f1-b84438c2e468 nodeName:}" failed. No retries permitted until 2026-07-28 12:12:59.835200686 +0000 UTC m=+593.652130496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls") pod "feast-simple-feast-remote-setup-74c5c767f9-t8ldm" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 28 12:12:59.335355 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:12:59.335224 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 12:12:59.335355 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:12:59.335294 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls podName:9c532a4a-1695-4db8-93f1-b84438c2e468 nodeName:}" failed. No retries permitted until 2026-07-28 12:12:59.83527635 +0000 UTC m=+593.652206151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls") pod "feast-simple-feast-remote-setup-74c5c767f9-t8ldm" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 12:12:59.335566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.335546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9c532a4a-1695-4db8-93f1-b84438c2e468-feast-data\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.335792 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.335773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9c532a4a-1695-4db8-93f1-b84438c2e468-registry-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.337634 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.337614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-ui-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.347717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.347687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9lnj\" (UniqueName: \"kubernetes.io/projected/9c532a4a-1695-4db8-93f1-b84438c2e468-kube-api-access-c9lnj\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.839571 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.839538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.839745 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.839601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.841928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.841907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.842053 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.841980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls\") pod \"feast-simple-feast-remote-setup-74c5c767f9-t8ldm\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:12:59.974109 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:12:59.974084 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:00.106075 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:00.106044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm"] Jul 28 12:13:00.107774 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:13:00.107753 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c532a4a_1695_4db8_93f1_b84438c2e468.slice/crio-c5b4a327a2ab6ce1417b04b75057baf9662728a79e06625897440a645320cee8 WatchSource:0}: Error finding container c5b4a327a2ab6ce1417b04b75057baf9662728a79e06625897440a645320cee8: Status 404 returned error can't find the container with id c5b4a327a2ab6ce1417b04b75057baf9662728a79e06625897440a645320cee8 Jul 28 12:13:00.429474 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:00.429389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerStarted","Data":"fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5"} Jul 28 12:13:00.429474 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:00.429443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerStarted","Data":"c5b4a327a2ab6ce1417b04b75057baf9662728a79e06625897440a645320cee8"} Jul 28 12:13:05.450956 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:05.450924 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerID="fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5" exitCode=0 Jul 28 12:13:05.451308 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:05.450964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerDied","Data":"fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5"} Jul 28 12:13:06.457839 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:06.457805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerStarted","Data":"5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1"} Jul 28 12:13:06.623463 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:06.623430 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:13:06.630336 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:06.630318 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:13:11.482294 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:11.482268 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerID="5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1" exitCode=0 Jul 28 12:13:11.482709 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:11.482350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerDied","Data":"5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1"} Jul 28 12:13:11.485947 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:11.485931 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 28 12:13:12.490536 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:12.490482 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerStarted","Data":"f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4"} Jul 28 12:13:12.491032 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:12.490562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerStarted","Data":"530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999"} Jul 28 12:13:12.491032 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:12.490585 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerStarted","Data":"53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668"} Jul 28 12:13:12.516935 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:12.516872 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podStartSLOduration=13.516856522 podStartE2EDuration="13.516856522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:13:12.514596895 +0000 UTC m=+606.331526712" watchObservedRunningTime="2026-07-28 12:13:12.516856522 +0000 UTC m=+606.333786339" Jul 28 12:13:14.975082 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:14.975042 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:14.975507 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:14.975112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:14.975507 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:14.975131 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:14.976386 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:14.976358 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Jul 28 12:13:14.976466 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:14.976388 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jul 28 12:13:14.976466 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:14.976404 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jul 28 12:13:17.976035 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:17.975998 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:17.976443 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:17.976148 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:17.976634 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:17.976578 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:17.976634 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:17.976622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:17.976912 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:17.976872 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:17.977021 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:17.977002 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:17.983080 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:17.983054 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:18.521189 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:18.521158 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:13:18.526021 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:13:18.525997 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:14:20.638045 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.638012 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm"] Jul 28 12:14:20.638868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.638236 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" containerID="cri-o://53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668" gracePeriod=30 Jul 28 12:14:20.638868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.638276 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" containerID="cri-o://f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4" gracePeriod=30 Jul 28 12:14:20.638868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.638278 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="offline" containerID="cri-o://530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999" gracePeriod=30 Jul 28 12:14:20.743721 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.743685 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf"] Jul 28 12:14:20.744014 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.743984 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="registry" containerID="cri-o://3745a06b17628b5490e6a52cc4a98bbee47f4a7eb8e677b11f03f82c45ca1277" gracePeriod=30 Jul 28 12:14:20.744176 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.744009 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" containerID="cri-o://fe25d239ce32d010cbce7b360affb661d1ddd753d43ccdcdff3f34a652e8e29a" gracePeriod=30 Jul 28 12:14:20.744176 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.744041 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="offline" containerID="cri-o://33290375be495a65f1fcbb43aaaf273e9cf45b5e1453bc66b68ec43e06b16d1d" gracePeriod=30 Jul 28 12:14:20.744176 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:20.744057 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" containerID="cri-o://3e2257549a97d93ab13467471c08bed13336eee83da5f1e66529e6572635609b" gracePeriod=30 Jul 28 12:14:21.788493 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.788469 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerID="f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4" exitCode=0 Jul 28 12:14:21.788493 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.788494 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerID="53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668" exitCode=0 Jul 28 12:14:21.788911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.788541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerDied","Data":"f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4"} Jul 28 12:14:21.788911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.788578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerDied","Data":"53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668"} Jul 28 12:14:21.791491 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.791472 2564 generic.go:356] "Generic (PLEG): container finished" podID="75dc0468-0375-4514-9821-8b42e225a3d5" containerID="fe25d239ce32d010cbce7b360affb661d1ddd753d43ccdcdff3f34a652e8e29a" exitCode=0 Jul 28 12:14:21.791491 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.791490 2564 generic.go:356] "Generic (PLEG): container finished" podID="75dc0468-0375-4514-9821-8b42e225a3d5" containerID="3e2257549a97d93ab13467471c08bed13336eee83da5f1e66529e6572635609b" exitCode=0 Jul 28 12:14:21.791612 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.791538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerDied","Data":"fe25d239ce32d010cbce7b360affb661d1ddd753d43ccdcdff3f34a652e8e29a"} Jul 28 12:14:21.791612 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:21.791567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerDied","Data":"3e2257549a97d93ab13467471c08bed13336eee83da5f1e66529e6572635609b"} Jul 28 12:14:26.153298 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:26.153261 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Jul 28 12:14:26.153697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:26.153299 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 28 12:14:27.977106 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:27.977056 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jul 28 12:14:28.521826 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:28.521790 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jul 28 12:14:36.153250 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:36.153207 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Jul 28 12:14:36.153641 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:36.153304 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 28 12:14:37.977434 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:37.977390 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jul 28 12:14:38.521233 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:38.521198 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jul 28 12:14:46.153706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:46.153663 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Jul 28 12:14:46.154111 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:46.153742 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Jul 28 12:14:46.154111 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:46.153826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:14:46.154111 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:46.153907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:14:47.977544 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:47.977492 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jul 28 12:14:47.978025 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:47.977643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:14:48.521799 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:48.521766 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jul 28 12:14:48.521984 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:48.521872 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:14:50.868116 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.868096 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:14:50.914688 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.914621 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerID="530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999" exitCode=137 Jul 28 12:14:50.914688 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.914673 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" Jul 28 12:14:50.914954 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.914784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerDied","Data":"530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999"} Jul 28 12:14:50.914954 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.914837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm" event={"ID":"9c532a4a-1695-4db8-93f1-b84438c2e468","Type":"ContainerDied","Data":"c5b4a327a2ab6ce1417b04b75057baf9662728a79e06625897440a645320cee8"} Jul 28 12:14:50.914954 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.914864 2564 scope.go:117] "RemoveContainer" containerID="f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4" Jul 28 12:14:50.918040 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.918017 2564 generic.go:356] "Generic (PLEG): container finished" podID="75dc0468-0375-4514-9821-8b42e225a3d5" containerID="33290375be495a65f1fcbb43aaaf273e9cf45b5e1453bc66b68ec43e06b16d1d" exitCode=137 Jul 28 12:14:50.918040 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.918039 2564 generic.go:356] "Generic (PLEG): container finished" podID="75dc0468-0375-4514-9821-8b42e225a3d5" containerID="3745a06b17628b5490e6a52cc4a98bbee47f4a7eb8e677b11f03f82c45ca1277" exitCode=137 Jul 28 12:14:50.918165 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.918082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerDied","Data":"33290375be495a65f1fcbb43aaaf273e9cf45b5e1453bc66b68ec43e06b16d1d"} Jul 28 12:14:50.918165 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.918113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerDied","Data":"3745a06b17628b5490e6a52cc4a98bbee47f4a7eb8e677b11f03f82c45ca1277"} Jul 28 12:14:50.923674 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.923660 2564 scope.go:117] "RemoveContainer" containerID="530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999" Jul 28 12:14:50.931283 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.931268 2564 scope.go:117] "RemoveContainer" containerID="53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668" Jul 28 12:14:50.938710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.938692 2564 scope.go:117] "RemoveContainer" containerID="5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1" Jul 28 12:14:50.945957 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.945943 2564 scope.go:117] "RemoveContainer" containerID="fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5" Jul 28 12:14:50.952730 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.952710 2564 scope.go:117] "RemoveContainer" containerID="f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4" Jul 28 12:14:50.952977 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:14:50.952962 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4\": container with ID starting with f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4 not found: ID does not exist" containerID="f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4" Jul 28 12:14:50.953035 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.952985 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4"} err="failed to get container status \"f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4\": rpc error: code = NotFound desc = could not find container \"f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4\": container with ID starting with f00d938c153beb1a67a1727213374f24bdb21fe36a4af59d3614039b71b197c4 not found: ID does not exist" Jul 28 12:14:50.953035 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.953002 2564 scope.go:117] "RemoveContainer" containerID="530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999" Jul 28 12:14:50.953257 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:14:50.953240 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999\": container with ID starting with 530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999 not found: ID does not exist" containerID="530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999" Jul 28 12:14:50.953305 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.953265 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999"} err="failed to get container status \"530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999\": rpc error: code = NotFound desc = could not find container \"530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999\": container with ID starting with 530a3033ef76c7262be3c1e5efc900ca8da563f01905fc2a598ca56016545999 not found: ID does not exist" Jul 28 12:14:50.953305 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.953284 2564 scope.go:117] "RemoveContainer" containerID="53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668" Jul 28 12:14:50.953539 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:14:50.953523 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668\": container with ID starting with 53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668 not found: ID does not exist" containerID="53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668" Jul 28 12:14:50.953583 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.953544 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668"} err="failed to get container status \"53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668\": rpc error: code = NotFound desc = could not find container \"53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668\": container with ID starting with 53741e05632b0befecfd1ce7baaedef2f7795b4cc00b35c56ed7b96eebd11668 not found: ID does not exist" Jul 28 12:14:50.953583 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.953559 2564 scope.go:117] "RemoveContainer" containerID="5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1" Jul 28 12:14:50.953776 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:14:50.953760 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1\": container with ID starting with 5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1 not found: ID does not exist" containerID="5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1" Jul 28 12:14:50.953826 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.953780 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1"} err="failed to get container status \"5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1\": rpc error: code = NotFound desc = could not find container \"5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1\": container with ID starting with 5a567cd4a30933a62cc2f1528d3dd1360bad4b2cde6dd7af0513f09838a0f4a1 not found: ID does not exist" Jul 28 12:14:50.953826 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.953793 2564 scope.go:117] "RemoveContainer" containerID="fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5" Jul 28 12:14:50.954012 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:14:50.953996 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5\": container with ID starting with fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5 not found: ID does not exist" containerID="fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5" Jul 28 12:14:50.954055 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:50.954015 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5"} err="failed to get container status \"fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5\": rpc error: code = NotFound desc = could not find container \"fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5\": container with ID starting with fab0596fe8e8580c21082f848b0525a7490a1028ad31f9c8a2f1083846ca65f5 not found: ID does not exist" Jul 28 12:14:51.000399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls\") pod \"9c532a4a-1695-4db8-93f1-b84438c2e468\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " Jul 28 12:14:51.000480 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000420 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9c532a4a-1695-4db8-93f1-b84438c2e468-feast-data\") pod \"9c532a4a-1695-4db8-93f1-b84438c2e468\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " Jul 28 12:14:51.000480 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000441 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9c532a4a-1695-4db8-93f1-b84438c2e468-registry-tls\") pod \"9c532a4a-1695-4db8-93f1-b84438c2e468\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " Jul 28 12:14:51.000548 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000483 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-ui-tls\") pod \"9c532a4a-1695-4db8-93f1-b84438c2e468\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " Jul 28 12:14:51.000548 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000503 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9lnj\" (UniqueName: \"kubernetes.io/projected/9c532a4a-1695-4db8-93f1-b84438c2e468-kube-api-access-c9lnj\") pod \"9c532a4a-1695-4db8-93f1-b84438c2e468\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " Jul 28 12:14:51.000643 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000549 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls\") pod \"9c532a4a-1695-4db8-93f1-b84438c2e468\" (UID: \"9c532a4a-1695-4db8-93f1-b84438c2e468\") " Jul 28 12:14:51.000867 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000840 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c532a4a-1695-4db8-93f1-b84438c2e468-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9c532a4a-1695-4db8-93f1-b84438c2e468" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:14:51.000979 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.000926 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c532a4a-1695-4db8-93f1-b84438c2e468-feast-data" (OuterVolumeSpecName: "feast-data") pod "9c532a4a-1695-4db8-93f1-b84438c2e468" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:14:51.002554 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.002527 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls" (OuterVolumeSpecName: "online-tls") pod "9c532a4a-1695-4db8-93f1-b84438c2e468" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:14:51.002643 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.002625 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c532a4a-1695-4db8-93f1-b84438c2e468-kube-api-access-c9lnj" (OuterVolumeSpecName: "kube-api-access-c9lnj") pod "9c532a4a-1695-4db8-93f1-b84438c2e468" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468"). InnerVolumeSpecName "kube-api-access-c9lnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:14:51.003043 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.003017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9c532a4a-1695-4db8-93f1-b84438c2e468" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:14:51.003043 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.003028 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9c532a4a-1695-4db8-93f1-b84438c2e468" (UID: "9c532a4a-1695-4db8-93f1-b84438c2e468"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:14:51.102107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.102067 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.102107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.102103 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9lnj\" (UniqueName: \"kubernetes.io/projected/9c532a4a-1695-4db8-93f1-b84438c2e468-kube-api-access-c9lnj\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.102107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.102112 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.102107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.102121 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9c532a4a-1695-4db8-93f1-b84438c2e468-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.102325 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.102129 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9c532a4a-1695-4db8-93f1-b84438c2e468-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.102325 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.102136 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/9c532a4a-1695-4db8-93f1-b84438c2e468-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.237939 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.237901 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm"] Jul 28 12:14:51.244176 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.244155 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-74c5c767f9-t8ldm"] Jul 28 12:14:51.395218 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.395197 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:14:51.505775 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.505717 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-offline-tls\") pod \"75dc0468-0375-4514-9821-8b42e225a3d5\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " Jul 28 12:14:51.505775 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.505749 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-ui-tls\") pod \"75dc0468-0375-4514-9821-8b42e225a3d5\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " Jul 28 12:14:51.506072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.505779 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhkmm\" (UniqueName: \"kubernetes.io/projected/75dc0468-0375-4514-9821-8b42e225a3d5-kube-api-access-mhkmm\") pod \"75dc0468-0375-4514-9821-8b42e225a3d5\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " Jul 28 12:14:51.506072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.505803 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-registry-tls\") pod \"75dc0468-0375-4514-9821-8b42e225a3d5\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " Jul 28 12:14:51.506072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.505856 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-online-tls\") pod \"75dc0468-0375-4514-9821-8b42e225a3d5\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " Jul 28 12:14:51.506072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.505922 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/75dc0468-0375-4514-9821-8b42e225a3d5-feast-data\") pod \"75dc0468-0375-4514-9821-8b42e225a3d5\" (UID: \"75dc0468-0375-4514-9821-8b42e225a3d5\") " Jul 28 12:14:51.506794 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.506764 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75dc0468-0375-4514-9821-8b42e225a3d5-feast-data" (OuterVolumeSpecName: "feast-data") pod "75dc0468-0375-4514-9821-8b42e225a3d5" (UID: "75dc0468-0375-4514-9821-8b42e225a3d5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:14:51.507926 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.507902 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75dc0468-0375-4514-9821-8b42e225a3d5-kube-api-access-mhkmm" (OuterVolumeSpecName: "kube-api-access-mhkmm") pod "75dc0468-0375-4514-9821-8b42e225a3d5" (UID: "75dc0468-0375-4514-9821-8b42e225a3d5"). InnerVolumeSpecName "kube-api-access-mhkmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:14:51.508029 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.507983 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "75dc0468-0375-4514-9821-8b42e225a3d5" (UID: "75dc0468-0375-4514-9821-8b42e225a3d5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:14:51.508082 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.508065 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-online-tls" (OuterVolumeSpecName: "online-tls") pod "75dc0468-0375-4514-9821-8b42e225a3d5" (UID: "75dc0468-0375-4514-9821-8b42e225a3d5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:14:51.508370 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.508345 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "75dc0468-0375-4514-9821-8b42e225a3d5" (UID: "75dc0468-0375-4514-9821-8b42e225a3d5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:14:51.508370 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.508360 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "75dc0468-0375-4514-9821-8b42e225a3d5" (UID: "75dc0468-0375-4514-9821-8b42e225a3d5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:14:51.607145 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.607108 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.607145 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.607141 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.607274 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.607154 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhkmm\" (UniqueName: \"kubernetes.io/projected/75dc0468-0375-4514-9821-8b42e225a3d5-kube-api-access-mhkmm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.607274 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.607167 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.607274 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.607179 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/75dc0468-0375-4514-9821-8b42e225a3d5-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.607274 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.607189 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/75dc0468-0375-4514-9821-8b42e225a3d5-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:14:51.929745 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.929664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" event={"ID":"75dc0468-0375-4514-9821-8b42e225a3d5","Type":"ContainerDied","Data":"95d95c28131e586e5294884c7ddf91a3e2cb575338e035716c7123eafa6532b9"} Jul 28 12:14:51.929745 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.929711 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf" Jul 28 12:14:51.930172 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.929748 2564 scope.go:117] "RemoveContainer" containerID="fe25d239ce32d010cbce7b360affb661d1ddd753d43ccdcdff3f34a652e8e29a" Jul 28 12:14:51.938324 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.938309 2564 scope.go:117] "RemoveContainer" containerID="33290375be495a65f1fcbb43aaaf273e9cf45b5e1453bc66b68ec43e06b16d1d" Jul 28 12:14:51.947457 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.947440 2564 scope.go:117] "RemoveContainer" containerID="3e2257549a97d93ab13467471c08bed13336eee83da5f1e66529e6572635609b" Jul 28 12:14:51.954594 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.954572 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf"] Jul 28 12:14:51.955399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.955384 2564 scope.go:117] "RemoveContainer" containerID="3745a06b17628b5490e6a52cc4a98bbee47f4a7eb8e677b11f03f82c45ca1277" Jul 28 12:14:51.959557 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.959537 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-5d594c7856-jhzxf"] Jul 28 12:14:51.963069 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.963049 2564 scope.go:117] "RemoveContainer" containerID="43cf4ea89491cc5948ce5d29156885cf2584c957276fdd0de39d5c51613fab9a" Jul 28 12:14:51.970129 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:51.970115 2564 scope.go:117] "RemoveContainer" containerID="3f74b314a6d4c5b07a8c378ce08f4b6f90c0907d1889f595c23133f9b5d212b9" Jul 28 12:14:52.780228 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:52.780192 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" path="/var/lib/kubelet/pods/75dc0468-0375-4514-9821-8b42e225a3d5/volumes" Jul 28 12:14:52.780798 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:14:52.780783 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" path="/var/lib/kubelet/pods/9c532a4a-1695-4db8-93f1-b84438c2e468/volumes" Jul 28 12:15:03.732374 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732300 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-nb855"] Jul 28 12:15:03.732804 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732787 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="feast-init" Jul 28 12:15:03.732804 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732806 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="feast-init" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732816 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732821 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732836 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732841 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732848 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="offline" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732852 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="offline" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732859 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="feast-init" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732864 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="feast-init" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732869 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="offline" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732895 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="offline" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732902 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732907 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" Jul 28 12:15:03.732911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732915 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="registry" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732920 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="registry" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732926 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732931 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732942 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="feast-apply" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732947 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="feast-apply" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732953 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="feast-apply" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.732959 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="feast-apply" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.733017 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="online" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.733024 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="offline" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.733030 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="online" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.733035 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c532a4a-1695-4db8-93f1-b84438c2e468" containerName="ui" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.733043 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="registry" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.733050 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="ui" Jul 28 12:15:03.733299 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.733057 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="75dc0468-0375-4514-9821-8b42e225a3d5" containerName="offline" Jul 28 12:15:03.735959 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.735945 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:03.738638 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.738617 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 12:15:03.738746 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.738670 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 12:15:03.739933 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.739914 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-mhp6m\"" Jul 28 12:15:03.739992 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.739937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 28 12:15:03.743655 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.743607 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-nb855"] Jul 28 12:15:03.801096 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.801068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6759004c-5bdb-4e47-a4b7-6d7934049e94-postgresdata\") pod \"postgres-864846d4f5-nb855\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:03.801219 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.801129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rcz\" (UniqueName: \"kubernetes.io/projected/6759004c-5bdb-4e47-a4b7-6d7934049e94-kube-api-access-z2rcz\") pod \"postgres-864846d4f5-nb855\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:03.902350 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.902327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rcz\" (UniqueName: \"kubernetes.io/projected/6759004c-5bdb-4e47-a4b7-6d7934049e94-kube-api-access-z2rcz\") pod \"postgres-864846d4f5-nb855\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:03.902447 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.902379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6759004c-5bdb-4e47-a4b7-6d7934049e94-postgresdata\") pod \"postgres-864846d4f5-nb855\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:03.902662 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.902649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6759004c-5bdb-4e47-a4b7-6d7934049e94-postgresdata\") pod \"postgres-864846d4f5-nb855\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:03.910957 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:03.910933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rcz\" (UniqueName: \"kubernetes.io/projected/6759004c-5bdb-4e47-a4b7-6d7934049e94-kube-api-access-z2rcz\") pod \"postgres-864846d4f5-nb855\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:04.055783 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:04.055725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:15:04.179717 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:04.179688 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-nb855"] Jul 28 12:15:04.182833 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:15:04.182809 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6759004c_5bdb_4e47_a4b7_6d7934049e94.slice/crio-828d64deb3246d8f5f52371451445773a537b785e8170bcd49cb5ae1eda80d01 WatchSource:0}: Error finding container 828d64deb3246d8f5f52371451445773a537b785e8170bcd49cb5ae1eda80d01: Status 404 returned error can't find the container with id 828d64deb3246d8f5f52371451445773a537b785e8170bcd49cb5ae1eda80d01 Jul 28 12:15:04.184698 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:04.184681 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:15:04.986255 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:04.986223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-nb855" event={"ID":"6759004c-5bdb-4e47-a4b7-6d7934049e94","Type":"ContainerStarted","Data":"828d64deb3246d8f5f52371451445773a537b785e8170bcd49cb5ae1eda80d01"} Jul 28 12:15:09.256271 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:09.256249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 28 12:15:10.008606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:10.008573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-nb855" event={"ID":"6759004c-5bdb-4e47-a4b7-6d7934049e94","Type":"ContainerStarted","Data":"3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1"} Jul 28 12:15:10.025690 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:10.025643 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-nb855" podStartSLOduration=1.956691057 podStartE2EDuration="7.025627734s" totalImagesPullingTime="5.068936677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:15:10.024950504 +0000 UTC m=+723.841880323" watchObservedRunningTime="2026-07-28 12:15:10.025627734 +0000 UTC m=+723.842557551" Jul 28 12:15:14.415276 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.415239 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2"] Jul 28 12:15:14.426619 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.426599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.429571 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.429546 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 28 12:15:14.429787 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.429761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 28 12:15:14.429863 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.429605 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 28 12:15:14.429944 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.429680 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-4dh9c\"" Jul 28 12:15:14.430009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.429651 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 28 12:15:14.432526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.432504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2"] Jul 28 12:15:14.590059 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.590024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.590059 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.590059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc0464-d85e-4b84-8e73-0d469e35e071-feast-data\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.590265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.590092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7l5\" (UniqueName: \"kubernetes.io/projected/02dc0464-d85e-4b84-8e73-0d469e35e071-kube-api-access-5x7l5\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.590265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.590161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-online-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.590265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.590198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-offline-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.691321 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.691245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.691321 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.691282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc0464-d85e-4b84-8e73-0d469e35e071-feast-data\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.691321 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.691313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7l5\" (UniqueName: \"kubernetes.io/projected/02dc0464-d85e-4b84-8e73-0d469e35e071-kube-api-access-5x7l5\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.691568 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.691345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-online-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.691568 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.691376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-offline-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.691568 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:15:14.691397 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 28 12:15:14.691568 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:15:14.691474 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls podName:02dc0464-d85e-4b84-8e73-0d469e35e071 nodeName:}" failed. No retries permitted until 2026-07-28 12:15:15.191454011 +0000 UTC m=+729.008383813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls") pod "feast-credit-scoring-5fdf8dbb6b-hw7t2" (UID: "02dc0464-d85e-4b84-8e73-0d469e35e071") : secret "feast-credit-scoring-registry-tls" not found Jul 28 12:15:14.691816 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.691629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc0464-d85e-4b84-8e73-0d469e35e071-feast-data\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.693942 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.693922 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-online-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.694135 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.694116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-offline-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:14.700453 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:14.700424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7l5\" (UniqueName: \"kubernetes.io/projected/02dc0464-d85e-4b84-8e73-0d469e35e071-kube-api-access-5x7l5\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:15.194866 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:15.194825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:15.197226 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:15.197181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls\") pod \"feast-credit-scoring-5fdf8dbb6b-hw7t2\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:15.336939 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:15.336915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:15.471273 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:15.471251 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2"] Jul 28 12:15:15.472985 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:15:15.472955 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dc0464_d85e_4b84_8e73_0d469e35e071.slice/crio-1541a464cc3c15c218d1239a21eda7a6bf3607dde0b82e3fa76872ea05e5b1ad WatchSource:0}: Error finding container 1541a464cc3c15c218d1239a21eda7a6bf3607dde0b82e3fa76872ea05e5b1ad: Status 404 returned error can't find the container with id 1541a464cc3c15c218d1239a21eda7a6bf3607dde0b82e3fa76872ea05e5b1ad Jul 28 12:15:16.033730 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:16.033695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerStarted","Data":"8b0d1be2d60961092ec32d3a5e0136430b9438e384bd879f0e69fa66e5da81b1"} Jul 28 12:15:16.033909 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:16.033746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerStarted","Data":"1541a464cc3c15c218d1239a21eda7a6bf3607dde0b82e3fa76872ea05e5b1ad"} Jul 28 12:15:17.038689 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:17.038655 2564 generic.go:356] "Generic (PLEG): container finished" podID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerID="8b0d1be2d60961092ec32d3a5e0136430b9438e384bd879f0e69fa66e5da81b1" exitCode=0 Jul 28 12:15:17.039015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:17.038742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerDied","Data":"8b0d1be2d60961092ec32d3a5e0136430b9438e384bd879f0e69fa66e5da81b1"} Jul 28 12:15:18.046323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:18.046291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerStarted","Data":"06f5f2ee084c6a91cc5f903bd353efd321266daa9f87a6a1c4445d7e5232b48d"} Jul 28 12:15:24.073147 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:24.073117 2564 generic.go:356] "Generic (PLEG): container finished" podID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerID="06f5f2ee084c6a91cc5f903bd353efd321266daa9f87a6a1c4445d7e5232b48d" exitCode=0 Jul 28 12:15:24.073566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:24.073172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerDied","Data":"06f5f2ee084c6a91cc5f903bd353efd321266daa9f87a6a1c4445d7e5232b48d"} Jul 28 12:15:25.082660 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:25.082626 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerStarted","Data":"a1f15223b6fc24adb142325dc5f260396d98f35930d6fa57ee1b02cb999659b2"} Jul 28 12:15:25.083157 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:25.082683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerStarted","Data":"2a1cb62fc1256ef9c5739ab606b233a91a1718013908ee12d92fc59933d369cd"} Jul 28 12:15:25.083157 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:25.082701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerStarted","Data":"b172ff251962454084c2cc29a85f819ea7a4722a68c1b05f2ec204a24976d62f"} Jul 28 12:15:25.130177 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:25.130131 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podStartSLOduration=11.130115835 podStartE2EDuration="11.130115835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:15:25.129530376 +0000 UTC m=+738.946460219" watchObservedRunningTime="2026-07-28 12:15:25.130115835 +0000 UTC m=+738.947045654" Jul 28 12:15:25.337079 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:25.336996 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:25.337221 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:25.337098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:25.337221 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:25.337152 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:27.337871 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:27.337836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:27.338282 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:27.338055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:27.338282 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:27.338086 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:27.339583 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:27.339542 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 12:15:27.339583 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:27.339550 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 28 12:15:27.339806 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:27.339550 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Jul 28 12:15:30.337483 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:30.337441 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 28 12:15:30.338438 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:30.338410 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:30.339388 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:30.339366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:30.342991 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:30.342974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:31.118435 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:31.118408 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:33.337849 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:33.337813 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 28 12:15:36.338178 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:36.338139 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:15:36.339023 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:15:36.339004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:16:39.992341 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:39.992308 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2"] Jul 28 12:16:39.992855 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:39.992546 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="registry" containerID="cri-o://b172ff251962454084c2cc29a85f819ea7a4722a68c1b05f2ec204a24976d62f" gracePeriod=30 Jul 28 12:16:39.992855 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:39.992625 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" containerID="cri-o://2a1cb62fc1256ef9c5739ab606b233a91a1718013908ee12d92fc59933d369cd" gracePeriod=30 Jul 28 12:16:39.992855 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:39.992620 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="offline" containerID="cri-o://a1f15223b6fc24adb142325dc5f260396d98f35930d6fa57ee1b02cb999659b2" gracePeriod=30 Jul 28 12:16:40.007619 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.007591 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-nb855"] Jul 28 12:16:40.007763 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.007746 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-nb855" podUID="6759004c-5bdb-4e47-a4b7-6d7934049e94" containerName="postgres" containerID="cri-o://3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1" gracePeriod=30 Jul 28 12:16:40.254799 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.254752 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:16:40.289199 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.289177 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2rcz\" (UniqueName: \"kubernetes.io/projected/6759004c-5bdb-4e47-a4b7-6d7934049e94-kube-api-access-z2rcz\") pod \"6759004c-5bdb-4e47-a4b7-6d7934049e94\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " Jul 28 12:16:40.289349 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.289220 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6759004c-5bdb-4e47-a4b7-6d7934049e94-postgresdata\") pod \"6759004c-5bdb-4e47-a4b7-6d7934049e94\" (UID: \"6759004c-5bdb-4e47-a4b7-6d7934049e94\") " Jul 28 12:16:40.291533 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.291448 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6759004c-5bdb-4e47-a4b7-6d7934049e94-kube-api-access-z2rcz" (OuterVolumeSpecName: "kube-api-access-z2rcz") pod "6759004c-5bdb-4e47-a4b7-6d7934049e94" (UID: "6759004c-5bdb-4e47-a4b7-6d7934049e94"). InnerVolumeSpecName "kube-api-access-z2rcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:16:40.320310 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.320282 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6759004c-5bdb-4e47-a4b7-6d7934049e94-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "6759004c-5bdb-4e47-a4b7-6d7934049e94" (UID: "6759004c-5bdb-4e47-a4b7-6d7934049e94"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:16:40.391049 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.391016 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2rcz\" (UniqueName: \"kubernetes.io/projected/6759004c-5bdb-4e47-a4b7-6d7934049e94-kube-api-access-z2rcz\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:16:40.391049 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.391051 2564 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6759004c-5bdb-4e47-a4b7-6d7934049e94-postgresdata\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:16:40.407560 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.407528 2564 generic.go:356] "Generic (PLEG): container finished" podID="6759004c-5bdb-4e47-a4b7-6d7934049e94" containerID="3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1" exitCode=0 Jul 28 12:16:40.407706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.407571 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-nb855" Jul 28 12:16:40.407706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.407570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-nb855" event={"ID":"6759004c-5bdb-4e47-a4b7-6d7934049e94","Type":"ContainerDied","Data":"3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1"} Jul 28 12:16:40.407706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.407668 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-nb855" event={"ID":"6759004c-5bdb-4e47-a4b7-6d7934049e94","Type":"ContainerDied","Data":"828d64deb3246d8f5f52371451445773a537b785e8170bcd49cb5ae1eda80d01"} Jul 28 12:16:40.407706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.407687 2564 scope.go:117] "RemoveContainer" containerID="3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1" Jul 28 12:16:40.416600 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.416576 2564 scope.go:117] "RemoveContainer" containerID="3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1" Jul 28 12:16:40.416866 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:16:40.416847 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1\": container with ID starting with 3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1 not found: ID does not exist" containerID="3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1" Jul 28 12:16:40.416953 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.416899 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1"} err="failed to get container status \"3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1\": rpc error: code = NotFound desc = could not find container \"3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1\": container with ID starting with 3fdab76a66590745089b2a4e852721b8c7117314aceb03b0957c944830bacda1 not found: ID does not exist" Jul 28 12:16:40.429805 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.429778 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-nb855"] Jul 28 12:16:40.436707 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.436684 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-nb855"] Jul 28 12:16:40.780148 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:40.780122 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6759004c-5bdb-4e47-a4b7-6d7934049e94" path="/var/lib/kubelet/pods/6759004c-5bdb-4e47-a4b7-6d7934049e94/volumes" Jul 28 12:16:41.115185 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:41.115155 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 12:16:41.414754 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:41.414684 2564 generic.go:356] "Generic (PLEG): container finished" podID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerID="2a1cb62fc1256ef9c5739ab606b233a91a1718013908ee12d92fc59933d369cd" exitCode=0 Jul 28 12:16:41.414899 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:41.414757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerDied","Data":"2a1cb62fc1256ef9c5739ab606b233a91a1718013908ee12d92fc59933d369cd"} Jul 28 12:16:51.116178 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:16:51.116137 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 12:17:01.115326 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:01.115288 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 12:17:01.115695 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:01.115407 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:17:10.339713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.339624 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Jul 28 12:17:10.541265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.541236 2564 generic.go:356] "Generic (PLEG): container finished" podID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerID="a1f15223b6fc24adb142325dc5f260396d98f35930d6fa57ee1b02cb999659b2" exitCode=137 Jul 28 12:17:10.541265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.541260 2564 generic.go:356] "Generic (PLEG): container finished" podID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerID="b172ff251962454084c2cc29a85f819ea7a4722a68c1b05f2ec204a24976d62f" exitCode=137 Jul 28 12:17:10.541495 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.541318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerDied","Data":"a1f15223b6fc24adb142325dc5f260396d98f35930d6fa57ee1b02cb999659b2"} Jul 28 12:17:10.541495 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.541372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerDied","Data":"b172ff251962454084c2cc29a85f819ea7a4722a68c1b05f2ec204a24976d62f"} Jul 28 12:17:10.641598 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.641576 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:17:10.719491 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.719459 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls\") pod \"02dc0464-d85e-4b84-8e73-0d469e35e071\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " Jul 28 12:17:10.719632 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.719516 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x7l5\" (UniqueName: \"kubernetes.io/projected/02dc0464-d85e-4b84-8e73-0d469e35e071-kube-api-access-5x7l5\") pod \"02dc0464-d85e-4b84-8e73-0d469e35e071\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " Jul 28 12:17:10.719632 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.719587 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-online-tls\") pod \"02dc0464-d85e-4b84-8e73-0d469e35e071\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " Jul 28 12:17:10.719758 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.719631 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc0464-d85e-4b84-8e73-0d469e35e071-feast-data\") pod \"02dc0464-d85e-4b84-8e73-0d469e35e071\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " Jul 28 12:17:10.719758 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.719663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-offline-tls\") pod \"02dc0464-d85e-4b84-8e73-0d469e35e071\" (UID: \"02dc0464-d85e-4b84-8e73-0d469e35e071\") " Jul 28 12:17:10.721909 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.721846 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-online-tls" (OuterVolumeSpecName: "online-tls") pod "02dc0464-d85e-4b84-8e73-0d469e35e071" (UID: "02dc0464-d85e-4b84-8e73-0d469e35e071"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:17:10.721909 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.721857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "02dc0464-d85e-4b84-8e73-0d469e35e071" (UID: "02dc0464-d85e-4b84-8e73-0d469e35e071"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:17:10.722080 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.722033 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "02dc0464-d85e-4b84-8e73-0d469e35e071" (UID: "02dc0464-d85e-4b84-8e73-0d469e35e071"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:17:10.722169 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.722149 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02dc0464-d85e-4b84-8e73-0d469e35e071-kube-api-access-5x7l5" (OuterVolumeSpecName: "kube-api-access-5x7l5") pod "02dc0464-d85e-4b84-8e73-0d469e35e071" (UID: "02dc0464-d85e-4b84-8e73-0d469e35e071"). InnerVolumeSpecName "kube-api-access-5x7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:17:10.727326 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.727302 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dc0464-d85e-4b84-8e73-0d469e35e071-feast-data" (OuterVolumeSpecName: "feast-data") pod "02dc0464-d85e-4b84-8e73-0d469e35e071" (UID: "02dc0464-d85e-4b84-8e73-0d469e35e071"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:17:10.820851 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.820826 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:17:10.820851 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.820851 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02dc0464-d85e-4b84-8e73-0d469e35e071-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:17:10.821015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.820864 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:17:10.821015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.820896 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02dc0464-d85e-4b84-8e73-0d469e35e071-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:17:10.821015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:10.820910 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x7l5\" (UniqueName: \"kubernetes.io/projected/02dc0464-d85e-4b84-8e73-0d469e35e071-kube-api-access-5x7l5\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:17:11.548482 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.548439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" event={"ID":"02dc0464-d85e-4b84-8e73-0d469e35e071","Type":"ContainerDied","Data":"1541a464cc3c15c218d1239a21eda7a6bf3607dde0b82e3fa76872ea05e5b1ad"} Jul 28 12:17:11.548951 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.548517 2564 scope.go:117] "RemoveContainer" containerID="a1f15223b6fc24adb142325dc5f260396d98f35930d6fa57ee1b02cb999659b2" Jul 28 12:17:11.548951 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.548454 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2" Jul 28 12:17:11.561769 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.561748 2564 scope.go:117] "RemoveContainer" containerID="2a1cb62fc1256ef9c5739ab606b233a91a1718013908ee12d92fc59933d369cd" Jul 28 12:17:11.568705 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.568677 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2"] Jul 28 12:17:11.570570 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.570557 2564 scope.go:117] "RemoveContainer" containerID="b172ff251962454084c2cc29a85f819ea7a4722a68c1b05f2ec204a24976d62f" Jul 28 12:17:11.573763 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.573740 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-5fdf8dbb6b-hw7t2"] Jul 28 12:17:11.578749 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.578734 2564 scope.go:117] "RemoveContainer" containerID="06f5f2ee084c6a91cc5f903bd353efd321266daa9f87a6a1c4445d7e5232b48d" Jul 28 12:17:11.585794 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:11.585779 2564 scope.go:117] "RemoveContainer" containerID="8b0d1be2d60961092ec32d3a5e0136430b9438e384bd879f0e69fa66e5da81b1" Jul 28 12:17:12.780017 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:17:12.779979 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" path="/var/lib/kubelet/pods/02dc0464-d85e-4b84-8e73-0d469e35e071/volumes" Jul 28 12:18:06.654021 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:18:06.653950 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:18:06.659783 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:18:06.659761 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:23:06.686061 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:06.686033 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:23:06.689374 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:06.689351 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:23:47.853993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.853963 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-8w8z6"] Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854301 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="registry" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854312 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="registry" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854324 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="feast-init" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854329 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="feast-init" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854345 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6759004c-5bdb-4e47-a4b7-6d7934049e94" containerName="postgres" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854351 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6759004c-5bdb-4e47-a4b7-6d7934049e94" containerName="postgres" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854361 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854369 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854380 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="feast-apply" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854384 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="feast-apply" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854390 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="offline" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854395 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="offline" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854445 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="online" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854454 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="offline" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854463 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02dc0464-d85e-4b84-8e73-0d469e35e071" containerName="registry" Jul 28 12:23:47.854582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.854470 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6759004c-5bdb-4e47-a4b7-6d7934049e94" containerName="postgres" Jul 28 12:23:47.857201 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.857186 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" Jul 28 12:23:47.862521 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.862494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 28 12:23:47.862813 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.862791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-dgrnn\"" Jul 28 12:23:47.862933 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.862791 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 28 12:23:47.883315 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.883291 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-8w8z6"] Jul 28 12:23:47.963461 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:47.963440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwfx\" (UniqueName: \"kubernetes.io/projected/124efc8d-3186-41da-9113-fdc2aa12151e-kube-api-access-5pwfx\") pod \"redis-69bbd798b6-8w8z6\" (UID: \"124efc8d-3186-41da-9113-fdc2aa12151e\") " pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" Jul 28 12:23:48.064652 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:48.064628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwfx\" (UniqueName: \"kubernetes.io/projected/124efc8d-3186-41da-9113-fdc2aa12151e-kube-api-access-5pwfx\") pod \"redis-69bbd798b6-8w8z6\" (UID: \"124efc8d-3186-41da-9113-fdc2aa12151e\") " pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" Jul 28 12:23:48.073045 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:48.073021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwfx\" (UniqueName: \"kubernetes.io/projected/124efc8d-3186-41da-9113-fdc2aa12151e-kube-api-access-5pwfx\") pod \"redis-69bbd798b6-8w8z6\" (UID: \"124efc8d-3186-41da-9113-fdc2aa12151e\") " pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" Jul 28 12:23:48.181593 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:48.181536 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" Jul 28 12:23:48.306724 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:48.306700 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-8w8z6"] Jul 28 12:23:48.309253 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:23:48.309222 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod124efc8d_3186_41da_9113_fdc2aa12151e.slice/crio-6d07983d0d8e3bb2f73acd369ca261d8048169c3d6141c07afe7c198b127e6f3 WatchSource:0}: Error finding container 6d07983d0d8e3bb2f73acd369ca261d8048169c3d6141c07afe7c198b127e6f3: Status 404 returned error can't find the container with id 6d07983d0d8e3bb2f73acd369ca261d8048169c3d6141c07afe7c198b127e6f3 Jul 28 12:23:48.311253 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:48.311235 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:23:49.149104 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:49.149075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" event={"ID":"124efc8d-3186-41da-9113-fdc2aa12151e","Type":"ContainerStarted","Data":"6d07983d0d8e3bb2f73acd369ca261d8048169c3d6141c07afe7c198b127e6f3"} Jul 28 12:23:50.154175 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:50.154095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" event={"ID":"124efc8d-3186-41da-9113-fdc2aa12151e","Type":"ContainerStarted","Data":"b18de28676ea21b6f1706b265f95681cdf41cfb89faa95248687cf00d42da24d"} Jul 28 12:23:50.169381 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:50.169337 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" podStartSLOduration=1.615190105 podStartE2EDuration="3.16932252s" totalImagesPullingTime="1.554132415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:23:50.168718429 +0000 UTC m=+1243.985648246" watchObservedRunningTime="2026-07-28 12:23:50.16932252 +0000 UTC m=+1243.986252336" Jul 28 12:23:53.415919 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.415872 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-msx2h"] Jul 28 12:23:53.419724 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.419707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.423101 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.423078 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 28 12:23:53.435320 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.435299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-msx2h"] Jul 28 12:23:53.504619 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.504597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kml2t\" (UniqueName: \"kubernetes.io/projected/6a5bf74c-d656-4e14-9b4e-a407ead1810f-kube-api-access-kml2t\") pod \"postgres-864846d4f5-msx2h\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.504718 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.504654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6a5bf74c-d656-4e14-9b4e-a407ead1810f-postgresdata\") pod \"postgres-864846d4f5-msx2h\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.605342 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.605319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6a5bf74c-d656-4e14-9b4e-a407ead1810f-postgresdata\") pod \"postgres-864846d4f5-msx2h\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.605443 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.605375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kml2t\" (UniqueName: \"kubernetes.io/projected/6a5bf74c-d656-4e14-9b4e-a407ead1810f-kube-api-access-kml2t\") pod \"postgres-864846d4f5-msx2h\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.605640 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.605625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6a5bf74c-d656-4e14-9b4e-a407ead1810f-postgresdata\") pod \"postgres-864846d4f5-msx2h\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.619335 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.619310 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kml2t\" (UniqueName: \"kubernetes.io/projected/6a5bf74c-d656-4e14-9b4e-a407ead1810f-kube-api-access-kml2t\") pod \"postgres-864846d4f5-msx2h\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.729245 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.729217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:23:53.863178 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:53.863152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-msx2h"] Jul 28 12:23:53.866104 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:23:53.866078 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a5bf74c_d656_4e14_9b4e_a407ead1810f.slice/crio-58f99fd9e5575e6ae948748563808fa170450d59da3a8d7c4ea60416c30ae767 WatchSource:0}: Error finding container 58f99fd9e5575e6ae948748563808fa170450d59da3a8d7c4ea60416c30ae767: Status 404 returned error can't find the container with id 58f99fd9e5575e6ae948748563808fa170450d59da3a8d7c4ea60416c30ae767 Jul 28 12:23:54.172615 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:54.172536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" event={"ID":"6a5bf74c-d656-4e14-9b4e-a407ead1810f","Type":"ContainerStarted","Data":"8a1575e0db8b7448c4efcfb014ec2c4b7874c32bf4b3bde1952961a3a4c649fe"} Jul 28 12:23:54.172615 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:54.172584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" event={"ID":"6a5bf74c-d656-4e14-9b4e-a407ead1810f","Type":"ContainerStarted","Data":"58f99fd9e5575e6ae948748563808fa170450d59da3a8d7c4ea60416c30ae767"} Jul 28 12:23:54.190488 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:54.190443 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" podStartSLOduration=1.190424521 podStartE2EDuration="1.190424521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:23:54.190021302 +0000 UTC m=+1248.006951120" watchObservedRunningTime="2026-07-28 12:23:54.190424521 +0000 UTC m=+1248.007354341" Jul 28 12:23:59.177684 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.177650 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t"] Jul 28 12:23:59.183427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.183407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.186066 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.186048 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 28 12:23:59.186182 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.186054 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-xkr7l\"" Jul 28 12:23:59.186182 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.186085 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 28 12:23:59.186470 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.186458 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 28 12:23:59.195771 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.195751 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t"] Jul 28 12:23:59.347032 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.347005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4b752459-3a04-4915-8ded-6ff13fb1f65d-feast-data\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.347137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.347055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-online-tls\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.347137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.347078 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sf96\" (UniqueName: \"kubernetes.io/projected/4b752459-3a04-4915-8ded-6ff13fb1f65d-kube-api-access-4sf96\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.347137 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.347131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.448379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.448302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.448379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.448376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4b752459-3a04-4915-8ded-6ff13fb1f65d-feast-data\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.448554 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.448436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-online-tls\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.448554 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:23:59.448446 2564 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 28 12:23:59.448554 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.448470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sf96\" (UniqueName: \"kubernetes.io/projected/4b752459-3a04-4915-8ded-6ff13fb1f65d-kube-api-access-4sf96\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.448554 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:23:59.448499 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls podName:4b752459-3a04-4915-8ded-6ff13fb1f65d nodeName:}" failed. No retries permitted until 2026-07-28 12:23:59.948482618 +0000 UTC m=+1253.765412414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls") pod "feast-credit-scoring-754b84db49-pjn6t" (UID: "4b752459-3a04-4915-8ded-6ff13fb1f65d") : secret "feast-credit-scoring-registry-tls" not found Jul 28 12:23:59.448763 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.448733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4b752459-3a04-4915-8ded-6ff13fb1f65d-feast-data\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.451314 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.451289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-online-tls\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.458323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.458299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sf96\" (UniqueName: \"kubernetes.io/projected/4b752459-3a04-4915-8ded-6ff13fb1f65d-kube-api-access-4sf96\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.953433 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.953404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:23:59.955941 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:23:59.955924 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls\") pod \"feast-credit-scoring-754b84db49-pjn6t\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:00.092586 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:00.092550 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:00.225817 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:00.225791 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t"] Jul 28 12:24:00.230087 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:24:00.230055 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b752459_3a04_4915_8ded_6ff13fb1f65d.slice/crio-97b70928d9dd94ca9319878058f79292df92c7054debc665e226c2e7f2a9d455 WatchSource:0}: Error finding container 97b70928d9dd94ca9319878058f79292df92c7054debc665e226c2e7f2a9d455: Status 404 returned error can't find the container with id 97b70928d9dd94ca9319878058f79292df92c7054debc665e226c2e7f2a9d455 Jul 28 12:24:01.203677 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:01.203643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerStarted","Data":"3ce357f41a51eb36b86f350958cdeacb329c0c1b8e259d1b2dd60ad0f8b65d54"} Jul 28 12:24:01.203839 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:01.203694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerStarted","Data":"97b70928d9dd94ca9319878058f79292df92c7054debc665e226c2e7f2a9d455"} Jul 28 12:24:02.208870 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:02.208843 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerID="3ce357f41a51eb36b86f350958cdeacb329c0c1b8e259d1b2dd60ad0f8b65d54" exitCode=0 Jul 28 12:24:02.209236 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:02.208932 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerDied","Data":"3ce357f41a51eb36b86f350958cdeacb329c0c1b8e259d1b2dd60ad0f8b65d54"} Jul 28 12:24:03.216315 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:03.216284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerStarted","Data":"05275d4103883e4946c512d29ea0ceb96a56325fd84b6b81aa556767bdf8d9f9"} Jul 28 12:24:09.244107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:09.244034 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerID="05275d4103883e4946c512d29ea0ceb96a56325fd84b6b81aa556767bdf8d9f9" exitCode=0 Jul 28 12:24:09.244107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:09.244087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerDied","Data":"05275d4103883e4946c512d29ea0ceb96a56325fd84b6b81aa556767bdf8d9f9"} Jul 28 12:24:09.247388 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:09.247369 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 28 12:24:10.251330 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:10.251298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerStarted","Data":"6c1bf00f8424c6453210ccc05cb7b40f8267175b856d10cdc3d0ddc9df7bd8f4"} Jul 28 12:24:10.251716 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:10.251357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerStarted","Data":"cfbbcba57f742b71cfe0e77290ddf83237c0eb47c3198c72b77744077f5c8da6"} Jul 28 12:24:10.278194 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:10.278138 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podStartSLOduration=11.278115551 podStartE2EDuration="11.278115551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:24:10.275750246 +0000 UTC m=+1264.092680102" watchObservedRunningTime="2026-07-28 12:24:10.278115551 +0000 UTC m=+1264.095045557" Jul 28 12:24:12.093162 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:12.093117 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:12.093546 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:12.093197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:12.094764 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:12.094737 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Jul 28 12:24:12.094913 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:12.094773 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 12:24:15.093389 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:15.093348 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Jul 28 12:24:15.354273 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:15.354197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:15.354793 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:15.354772 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:15.357497 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:15.357474 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:18.093212 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:18.093172 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Jul 28 12:24:20.093668 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:20.093633 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:21.093841 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:21.093813 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:21.094530 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:21.094510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:24:25.294494 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.294458 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv"] Jul 28 12:24:25.299353 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.299333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.302558 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.302529 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 28 12:24:25.302705 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.302672 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-q9qs9\"" Jul 28 12:24:25.302801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.302785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 28 12:24:25.303084 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.303066 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 28 12:24:25.310505 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.310485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv"] Jul 28 12:24:25.366168 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.366145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38955073-f668-4d0f-a0b8-70e45a1d1ef7-feast-data\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.366293 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.366189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp7t\" (UniqueName: \"kubernetes.io/projected/38955073-f668-4d0f-a0b8-70e45a1d1ef7-kube-api-access-xfp7t\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.366293 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.366243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-registry-tls\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.366293 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.366273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-online-tls\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.467550 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.467529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38955073-f668-4d0f-a0b8-70e45a1d1ef7-feast-data\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.467671 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.467575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp7t\" (UniqueName: \"kubernetes.io/projected/38955073-f668-4d0f-a0b8-70e45a1d1ef7-kube-api-access-xfp7t\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.467671 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.467625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-registry-tls\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.467671 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.467649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-online-tls\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.468019 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.467986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38955073-f668-4d0f-a0b8-70e45a1d1ef7-feast-data\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.470075 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.470057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-registry-tls\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.470195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.470179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-online-tls\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.476148 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.476130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp7t\" (UniqueName: \"kubernetes.io/projected/38955073-f668-4d0f-a0b8-70e45a1d1ef7-kube-api-access-xfp7t\") pod \"feast-driver-ranking-5f65df7666-8cddv\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.608712 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.608658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:25.741275 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:25.741247 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv"] Jul 28 12:24:25.743148 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:24:25.743122 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38955073_f668_4d0f_a0b8_70e45a1d1ef7.slice/crio-039f94255f7490a6239f76ce8554f2335a507e9ead7bc2c068be1d2c9c93fd00 WatchSource:0}: Error finding container 039f94255f7490a6239f76ce8554f2335a507e9ead7bc2c068be1d2c9c93fd00: Status 404 returned error can't find the container with id 039f94255f7490a6239f76ce8554f2335a507e9ead7bc2c068be1d2c9c93fd00 Jul 28 12:24:26.325952 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:26.325919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerStarted","Data":"44ddc5ca5f40682a0e2ca2bcd7763e2a6a01a1331568e2a6ae68ab1eb4a14c03"} Jul 28 12:24:26.326405 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:26.325974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerStarted","Data":"039f94255f7490a6239f76ce8554f2335a507e9ead7bc2c068be1d2c9c93fd00"} Jul 28 12:24:31.346615 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:31.346580 2564 generic.go:356] "Generic (PLEG): container finished" podID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerID="44ddc5ca5f40682a0e2ca2bcd7763e2a6a01a1331568e2a6ae68ab1eb4a14c03" exitCode=0 Jul 28 12:24:31.347008 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:31.346635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerDied","Data":"44ddc5ca5f40682a0e2ca2bcd7763e2a6a01a1331568e2a6ae68ab1eb4a14c03"} Jul 28 12:24:32.353949 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:32.353911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerStarted","Data":"bf6d679565d1c3e329295ef7c6c9d0b9126cb166bd1d504745df446759ed9532"} Jul 28 12:24:38.382502 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:38.382468 2564 generic.go:356] "Generic (PLEG): container finished" podID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerID="bf6d679565d1c3e329295ef7c6c9d0b9126cb166bd1d504745df446759ed9532" exitCode=0 Jul 28 12:24:38.382907 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:38.382541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerDied","Data":"bf6d679565d1c3e329295ef7c6c9d0b9126cb166bd1d504745df446759ed9532"} Jul 28 12:24:39.389485 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:39.389448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerStarted","Data":"e8ff104a6c4098ce888ca1480dab9f8dd488020821a3a49ee6b0393696850ec4"} Jul 28 12:24:39.389847 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:39.389526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerStarted","Data":"dbb42471e2015a297e3b44bb38a307094ca4626fd2ac5fb280322174dca8d973"} Jul 28 12:24:39.414406 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:39.414357 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podStartSLOduration=14.414343317 podStartE2EDuration="14.414343317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:24:39.412074551 +0000 UTC m=+1293.229004370" watchObservedRunningTime="2026-07-28 12:24:39.414343317 +0000 UTC m=+1293.231273134" Jul 28 12:24:40.609055 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:40.608987 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:40.609497 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:40.609107 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:40.610285 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:40.610253 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Jul 28 12:24:40.610285 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:40.610277 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 28 12:24:43.609191 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:43.609143 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Jul 28 12:24:43.609571 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:43.609149 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 28 12:24:45.609426 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:45.609393 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:45.609900 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:45.609459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:46.609592 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:46.609553 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Jul 28 12:24:46.615441 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:46.615419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:46.618842 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:46.618821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:49.609395 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:49.609369 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:24:49.610243 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:24:49.610219 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:26:06.560922 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:06.560893 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t"] Jul 28 12:26:06.561452 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:06.561091 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="registry" containerID="cri-o://cfbbcba57f742b71cfe0e77290ddf83237c0eb47c3198c72b77744077f5c8da6" gracePeriod=30 Jul 28 12:26:06.561452 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:06.561140 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" containerID="cri-o://6c1bf00f8424c6453210ccc05cb7b40f8267175b856d10cdc3d0ddc9df7bd8f4" gracePeriod=30 Jul 28 12:26:06.564848 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:06.564823 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv"] Jul 28 12:26:06.565055 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:06.565035 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="registry" containerID="cri-o://dbb42471e2015a297e3b44bb38a307094ca4626fd2ac5fb280322174dca8d973" gracePeriod=30 Jul 28 12:26:06.565115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:06.565079 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" containerID="cri-o://e8ff104a6c4098ce888ca1480dab9f8dd488020821a3a49ee6b0393696850ec4" gracePeriod=30 Jul 28 12:26:06.616117 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:06.616094 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 28 12:26:07.629140 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.629066 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-msx2h"] Jul 28 12:26:07.629523 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.629230 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" podUID="6a5bf74c-d656-4e14-9b4e-a407ead1810f" containerName="postgres" containerID="cri-o://8a1575e0db8b7448c4efcfb014ec2c4b7874c32bf4b3bde1952961a3a4c649fe" gracePeriod=30 Jul 28 12:26:07.639562 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.639538 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-8w8z6"] Jul 28 12:26:07.639730 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.639698 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" podUID="124efc8d-3186-41da-9113-fdc2aa12151e" containerName="redis" containerID="cri-o://b18de28676ea21b6f1706b265f95681cdf41cfb89faa95248687cf00d42da24d" gracePeriod=30 Jul 28 12:26:07.781093 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.781059 2564 generic.go:356] "Generic (PLEG): container finished" podID="124efc8d-3186-41da-9113-fdc2aa12151e" containerID="b18de28676ea21b6f1706b265f95681cdf41cfb89faa95248687cf00d42da24d" exitCode=0 Jul 28 12:26:07.781267 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.781126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" event={"ID":"124efc8d-3186-41da-9113-fdc2aa12151e","Type":"ContainerDied","Data":"b18de28676ea21b6f1706b265f95681cdf41cfb89faa95248687cf00d42da24d"} Jul 28 12:26:07.782764 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.782742 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a5bf74c-d656-4e14-9b4e-a407ead1810f" containerID="8a1575e0db8b7448c4efcfb014ec2c4b7874c32bf4b3bde1952961a3a4c649fe" exitCode=0 Jul 28 12:26:07.782908 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.782811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" event={"ID":"6a5bf74c-d656-4e14-9b4e-a407ead1810f","Type":"ContainerDied","Data":"8a1575e0db8b7448c4efcfb014ec2c4b7874c32bf4b3bde1952961a3a4c649fe"} Jul 28 12:26:07.785472 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.785272 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerID="6c1bf00f8424c6453210ccc05cb7b40f8267175b856d10cdc3d0ddc9df7bd8f4" exitCode=0 Jul 28 12:26:07.785472 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.785385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerDied","Data":"6c1bf00f8424c6453210ccc05cb7b40f8267175b856d10cdc3d0ddc9df7bd8f4"} Jul 28 12:26:07.788389 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.788371 2564 generic.go:356] "Generic (PLEG): container finished" podID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerID="e8ff104a6c4098ce888ca1480dab9f8dd488020821a3a49ee6b0393696850ec4" exitCode=0 Jul 28 12:26:07.788484 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.788434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerDied","Data":"e8ff104a6c4098ce888ca1480dab9f8dd488020821a3a49ee6b0393696850ec4"} Jul 28 12:26:07.886606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.886558 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" Jul 28 12:26:07.900799 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:07.900780 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:26:08.000407 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.000378 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pwfx\" (UniqueName: \"kubernetes.io/projected/124efc8d-3186-41da-9113-fdc2aa12151e-kube-api-access-5pwfx\") pod \"124efc8d-3186-41da-9113-fdc2aa12151e\" (UID: \"124efc8d-3186-41da-9113-fdc2aa12151e\") " Jul 28 12:26:08.000552 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.000462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kml2t\" (UniqueName: \"kubernetes.io/projected/6a5bf74c-d656-4e14-9b4e-a407ead1810f-kube-api-access-kml2t\") pod \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " Jul 28 12:26:08.000552 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.000497 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6a5bf74c-d656-4e14-9b4e-a407ead1810f-postgresdata\") pod \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\" (UID: \"6a5bf74c-d656-4e14-9b4e-a407ead1810f\") " Jul 28 12:26:08.002670 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.002636 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124efc8d-3186-41da-9113-fdc2aa12151e-kube-api-access-5pwfx" (OuterVolumeSpecName: "kube-api-access-5pwfx") pod "124efc8d-3186-41da-9113-fdc2aa12151e" (UID: "124efc8d-3186-41da-9113-fdc2aa12151e"). InnerVolumeSpecName "kube-api-access-5pwfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:26:08.002776 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.002686 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5bf74c-d656-4e14-9b4e-a407ead1810f-kube-api-access-kml2t" (OuterVolumeSpecName: "kube-api-access-kml2t") pod "6a5bf74c-d656-4e14-9b4e-a407ead1810f" (UID: "6a5bf74c-d656-4e14-9b4e-a407ead1810f"). InnerVolumeSpecName "kube-api-access-kml2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:26:08.030009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.029987 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5bf74c-d656-4e14-9b4e-a407ead1810f-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "6a5bf74c-d656-4e14-9b4e-a407ead1810f" (UID: "6a5bf74c-d656-4e14-9b4e-a407ead1810f"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:26:08.101223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.101189 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kml2t\" (UniqueName: \"kubernetes.io/projected/6a5bf74c-d656-4e14-9b4e-a407ead1810f-kube-api-access-kml2t\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:08.101223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.101222 2564 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6a5bf74c-d656-4e14-9b4e-a407ead1810f-postgresdata\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:08.101223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.101231 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwfx\" (UniqueName: \"kubernetes.io/projected/124efc8d-3186-41da-9113-fdc2aa12151e-kube-api-access-5pwfx\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:08.793570 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.793550 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" Jul 28 12:26:08.793570 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.793553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-8w8z6" event={"ID":"124efc8d-3186-41da-9113-fdc2aa12151e","Type":"ContainerDied","Data":"6d07983d0d8e3bb2f73acd369ca261d8048169c3d6141c07afe7c198b127e6f3"} Jul 28 12:26:08.794010 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.793594 2564 scope.go:117] "RemoveContainer" containerID="b18de28676ea21b6f1706b265f95681cdf41cfb89faa95248687cf00d42da24d" Jul 28 12:26:08.794950 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.794925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" event={"ID":"6a5bf74c-d656-4e14-9b4e-a407ead1810f","Type":"ContainerDied","Data":"58f99fd9e5575e6ae948748563808fa170450d59da3a8d7c4ea60416c30ae767"} Jul 28 12:26:08.795089 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.794939 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-msx2h" Jul 28 12:26:08.801964 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.801925 2564 scope.go:117] "RemoveContainer" containerID="8a1575e0db8b7448c4efcfb014ec2c4b7874c32bf4b3bde1952961a3a4c649fe" Jul 28 12:26:08.810594 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.810578 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-msx2h"] Jul 28 12:26:08.813255 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.813236 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-msx2h"] Jul 28 12:26:08.823165 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.823137 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-8w8z6"] Jul 28 12:26:08.828999 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:08.828981 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-8w8z6"] Jul 28 12:26:10.779659 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:10.779629 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124efc8d-3186-41da-9113-fdc2aa12151e" path="/var/lib/kubelet/pods/124efc8d-3186-41da-9113-fdc2aa12151e/volumes" Jul 28 12:26:10.780025 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:10.779968 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5bf74c-d656-4e14-9b4e-a407ead1810f" path="/var/lib/kubelet/pods/6a5bf74c-d656-4e14-9b4e-a407ead1810f/volumes" Jul 28 12:26:15.354923 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:15.354864 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 12:26:16.616510 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:16.616472 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 28 12:26:25.354897 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:25.354833 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 12:26:26.616194 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:26.616156 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 28 12:26:26.616566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:26.616298 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:26:35.355666 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:35.355626 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 12:26:35.356073 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:35.355756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:26:36.616921 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:36.616867 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 28 12:26:36.925384 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:36.925314 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerID="cfbbcba57f742b71cfe0e77290ddf83237c0eb47c3198c72b77744077f5c8da6" exitCode=137 Jul 28 12:26:36.925526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:36.925387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerDied","Data":"cfbbcba57f742b71cfe0e77290ddf83237c0eb47c3198c72b77744077f5c8da6"} Jul 28 12:26:36.927709 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:36.927689 2564 generic.go:356] "Generic (PLEG): container finished" podID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerID="dbb42471e2015a297e3b44bb38a307094ca4626fd2ac5fb280322174dca8d973" exitCode=137 Jul 28 12:26:36.927820 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:36.927733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerDied","Data":"dbb42471e2015a297e3b44bb38a307094ca4626fd2ac5fb280322174dca8d973"} Jul 28 12:26:37.226980 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.226959 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:26:37.229357 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.229335 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:26:37.316561 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316536 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls\") pod \"4b752459-3a04-4915-8ded-6ff13fb1f65d\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " Jul 28 12:26:37.316711 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316576 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38955073-f668-4d0f-a0b8-70e45a1d1ef7-feast-data\") pod \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " Jul 28 12:26:37.316711 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316623 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfp7t\" (UniqueName: \"kubernetes.io/projected/38955073-f668-4d0f-a0b8-70e45a1d1ef7-kube-api-access-xfp7t\") pod \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " Jul 28 12:26:37.316711 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316653 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-registry-tls\") pod \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " Jul 28 12:26:37.316711 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-online-tls\") pod \"4b752459-3a04-4915-8ded-6ff13fb1f65d\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " Jul 28 12:26:37.316985 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sf96\" (UniqueName: \"kubernetes.io/projected/4b752459-3a04-4915-8ded-6ff13fb1f65d-kube-api-access-4sf96\") pod \"4b752459-3a04-4915-8ded-6ff13fb1f65d\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " Jul 28 12:26:37.316985 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316742 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-online-tls\") pod \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\" (UID: \"38955073-f668-4d0f-a0b8-70e45a1d1ef7\") " Jul 28 12:26:37.316985 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.316777 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4b752459-3a04-4915-8ded-6ff13fb1f65d-feast-data\") pod \"4b752459-3a04-4915-8ded-6ff13fb1f65d\" (UID: \"4b752459-3a04-4915-8ded-6ff13fb1f65d\") " Jul 28 12:26:37.317240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.317197 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38955073-f668-4d0f-a0b8-70e45a1d1ef7-feast-data" (OuterVolumeSpecName: "feast-data") pod "38955073-f668-4d0f-a0b8-70e45a1d1ef7" (UID: "38955073-f668-4d0f-a0b8-70e45a1d1ef7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:26:37.319620 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.319586 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-online-tls" (OuterVolumeSpecName: "online-tls") pod "4b752459-3a04-4915-8ded-6ff13fb1f65d" (UID: "4b752459-3a04-4915-8ded-6ff13fb1f65d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:26:37.319620 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.319605 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4b752459-3a04-4915-8ded-6ff13fb1f65d" (UID: "4b752459-3a04-4915-8ded-6ff13fb1f65d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:26:37.319757 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.319624 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b752459-3a04-4915-8ded-6ff13fb1f65d-kube-api-access-4sf96" (OuterVolumeSpecName: "kube-api-access-4sf96") pod "4b752459-3a04-4915-8ded-6ff13fb1f65d" (UID: "4b752459-3a04-4915-8ded-6ff13fb1f65d"). InnerVolumeSpecName "kube-api-access-4sf96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:26:37.319757 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.319629 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-online-tls" (OuterVolumeSpecName: "online-tls") pod "38955073-f668-4d0f-a0b8-70e45a1d1ef7" (UID: "38955073-f668-4d0f-a0b8-70e45a1d1ef7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:26:37.319757 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.319660 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "38955073-f668-4d0f-a0b8-70e45a1d1ef7" (UID: "38955073-f668-4d0f-a0b8-70e45a1d1ef7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:26:37.319757 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.319688 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38955073-f668-4d0f-a0b8-70e45a1d1ef7-kube-api-access-xfp7t" (OuterVolumeSpecName: "kube-api-access-xfp7t") pod "38955073-f668-4d0f-a0b8-70e45a1d1ef7" (UID: "38955073-f668-4d0f-a0b8-70e45a1d1ef7"). InnerVolumeSpecName "kube-api-access-xfp7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:26:37.326378 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.326358 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b752459-3a04-4915-8ded-6ff13fb1f65d-feast-data" (OuterVolumeSpecName: "feast-data") pod "4b752459-3a04-4915-8ded-6ff13fb1f65d" (UID: "4b752459-3a04-4915-8ded-6ff13fb1f65d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:26:37.418045 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418011 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sf96\" (UniqueName: \"kubernetes.io/projected/4b752459-3a04-4915-8ded-6ff13fb1f65d-kube-api-access-4sf96\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.418045 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418038 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.418045 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418050 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4b752459-3a04-4915-8ded-6ff13fb1f65d-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.418264 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418061 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.418264 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418073 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/38955073-f668-4d0f-a0b8-70e45a1d1ef7-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.418264 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418083 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfp7t\" (UniqueName: \"kubernetes.io/projected/38955073-f668-4d0f-a0b8-70e45a1d1ef7-kube-api-access-xfp7t\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.418264 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418094 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/38955073-f668-4d0f-a0b8-70e45a1d1ef7-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.418264 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.418105 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4b752459-3a04-4915-8ded-6ff13fb1f65d-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:26:37.934747 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.934679 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" Jul 28 12:26:37.934747 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.934693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t" event={"ID":"4b752459-3a04-4915-8ded-6ff13fb1f65d","Type":"ContainerDied","Data":"97b70928d9dd94ca9319878058f79292df92c7054debc665e226c2e7f2a9d455"} Jul 28 12:26:37.935372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.934761 2564 scope.go:117] "RemoveContainer" containerID="6c1bf00f8424c6453210ccc05cb7b40f8267175b856d10cdc3d0ddc9df7bd8f4" Jul 28 12:26:37.937536 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.937512 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" event={"ID":"38955073-f668-4d0f-a0b8-70e45a1d1ef7","Type":"ContainerDied","Data":"039f94255f7490a6239f76ce8554f2335a507e9ead7bc2c068be1d2c9c93fd00"} Jul 28 12:26:37.937708 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.937551 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv" Jul 28 12:26:37.945835 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.945811 2564 scope.go:117] "RemoveContainer" containerID="cfbbcba57f742b71cfe0e77290ddf83237c0eb47c3198c72b77744077f5c8da6" Jul 28 12:26:37.955129 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.955114 2564 scope.go:117] "RemoveContainer" containerID="05275d4103883e4946c512d29ea0ceb96a56325fd84b6b81aa556767bdf8d9f9" Jul 28 12:26:37.962838 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.962816 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t"] Jul 28 12:26:37.963501 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.963478 2564 scope.go:117] "RemoveContainer" containerID="3ce357f41a51eb36b86f350958cdeacb329c0c1b8e259d1b2dd60ad0f8b65d54" Jul 28 12:26:37.965055 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.965036 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-754b84db49-pjn6t"] Jul 28 12:26:37.974064 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.974046 2564 scope.go:117] "RemoveContainer" containerID="e8ff104a6c4098ce888ca1480dab9f8dd488020821a3a49ee6b0393696850ec4" Jul 28 12:26:37.977576 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.977553 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv"] Jul 28 12:26:37.981428 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.981408 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5f65df7666-8cddv"] Jul 28 12:26:37.982200 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.982180 2564 scope.go:117] "RemoveContainer" containerID="dbb42471e2015a297e3b44bb38a307094ca4626fd2ac5fb280322174dca8d973" Jul 28 12:26:37.989629 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.989614 2564 scope.go:117] "RemoveContainer" containerID="bf6d679565d1c3e329295ef7c6c9d0b9126cb166bd1d504745df446759ed9532" Jul 28 12:26:37.996648 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:37.996631 2564 scope.go:117] "RemoveContainer" containerID="44ddc5ca5f40682a0e2ca2bcd7763e2a6a01a1331568e2a6ae68ab1eb4a14c03" Jul 28 12:26:38.779757 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:38.779729 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" path="/var/lib/kubelet/pods/38955073-f668-4d0f-a0b8-70e45a1d1ef7/volumes" Jul 28 12:26:38.780222 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:38.780208 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" path="/var/lib/kubelet/pods/4b752459-3a04-4915-8ded-6ff13fb1f65d/volumes" Jul 28 12:26:55.717914 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.717860 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn"] Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718187 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="feast-init" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718197 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="feast-init" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718209 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="registry" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718214 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="registry" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718221 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718226 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718235 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="registry" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718240 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="registry" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718250 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124efc8d-3186-41da-9113-fdc2aa12151e" containerName="redis" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718255 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="124efc8d-3186-41da-9113-fdc2aa12151e" containerName="redis" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718263 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="feast-apply" Jul 28 12:26:55.718266 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718267 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="feast-apply" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718276 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5bf74c-d656-4e14-9b4e-a407ead1810f" containerName="postgres" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718281 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5bf74c-d656-4e14-9b4e-a407ead1810f" containerName="postgres" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718288 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="feast-init" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718293 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="feast-init" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718303 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="feast-apply" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718307 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="feast-apply" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718317 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718321 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718399 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="registry" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718409 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5bf74c-d656-4e14-9b4e-a407ead1810f" containerName="postgres" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718417 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="registry" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718421 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="38955073-f668-4d0f-a0b8-70e45a1d1ef7" containerName="online" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718427 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="124efc8d-3186-41da-9113-fdc2aa12151e" containerName="redis" Jul 28 12:26:55.718679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.718432 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b752459-3a04-4915-8ded-6ff13fb1f65d" containerName="online" Jul 28 12:26:55.722760 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.722745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.725738 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.725721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 28 12:26:55.726406 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.726384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 28 12:26:55.726512 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.726426 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 12:26:55.726799 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.726778 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 28 12:26:55.726852 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.726802 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 12:26:55.726964 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.726941 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-k8w88\"" Jul 28 12:26:55.727052 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.726993 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 28 12:26:55.742504 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.742483 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn"] Jul 28 12:26:55.748302 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.748284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/419220c0-6ef6-4f01-a93b-3302a5d52207-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.748392 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.748318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.748392 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.748339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.748392 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.748371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzzv\" (UniqueName: \"kubernetes.io/projected/419220c0-6ef6-4f01-a93b-3302a5d52207-kube-api-access-lzzzv\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.748524 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.748400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.748524 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.748431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.849028 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.849003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.849154 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.849035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/419220c0-6ef6-4f01-a93b-3302a5d52207-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.849154 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.849060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.849154 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.849080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.849154 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.849103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzzv\" (UniqueName: \"kubernetes.io/projected/419220c0-6ef6-4f01-a93b-3302a5d52207-kube-api-access-lzzzv\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.849154 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.849131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.849436 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:26:55.849173 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 28 12:26:55.849436 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:26:55.849248 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls podName:419220c0-6ef6-4f01-a93b-3302a5d52207 nodeName:}" failed. No retries permitted until 2026-07-28 12:26:56.34922844 +0000 UTC m=+1430.166158241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls") pod "feast-simple-feast-setup-777d6cc998-7fvhn" (UID: "419220c0-6ef6-4f01-a93b-3302a5d52207") : secret "feast-simple-feast-setup-registry-tls" not found Jul 28 12:26:55.849566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.849436 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/419220c0-6ef6-4f01-a93b-3302a5d52207-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.851629 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.851609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.851718 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.851683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.851757 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.851683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:55.858944 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:55.858925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzzv\" (UniqueName: \"kubernetes.io/projected/419220c0-6ef6-4f01-a93b-3302a5d52207-kube-api-access-lzzzv\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:56.353864 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:56.353837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:56.356274 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:56.356247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-7fvhn\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:56.645265 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:56.645205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:26:56.782157 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:56.782135 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn"] Jul 28 12:26:56.783337 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:26:56.783314 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod419220c0_6ef6_4f01_a93b_3302a5d52207.slice/crio-362df739306e00477defd8db989894605a9f8f95fd3dbc108fe628d2be44067c WatchSource:0}: Error finding container 362df739306e00477defd8db989894605a9f8f95fd3dbc108fe628d2be44067c: Status 404 returned error can't find the container with id 362df739306e00477defd8db989894605a9f8f95fd3dbc108fe628d2be44067c Jul 28 12:26:57.016010 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:26:57.015983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerStarted","Data":"362df739306e00477defd8db989894605a9f8f95fd3dbc108fe628d2be44067c"} Jul 28 12:27:10.084895 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:10.084794 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerStarted","Data":"97bdfe48b878ffd29a25736f078eee3fca282ae798dc6eaa302abe7d56a4ffb1"} Jul 28 12:27:15.108774 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:15.108737 2564 generic.go:356] "Generic (PLEG): container finished" podID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerID="97bdfe48b878ffd29a25736f078eee3fca282ae798dc6eaa302abe7d56a4ffb1" exitCode=0 Jul 28 12:27:15.109175 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:15.108781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerDied","Data":"97bdfe48b878ffd29a25736f078eee3fca282ae798dc6eaa302abe7d56a4ffb1"} Jul 28 12:27:16.114177 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:16.114144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerStarted","Data":"dc438ccf941d4bab27d6029bdb44e7fc7287633a446ea1bc189464af9fd03762"} Jul 28 12:27:21.138344 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:21.138310 2564 generic.go:356] "Generic (PLEG): container finished" podID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerID="dc438ccf941d4bab27d6029bdb44e7fc7287633a446ea1bc189464af9fd03762" exitCode=0 Jul 28 12:27:21.138706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:21.138352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerDied","Data":"dc438ccf941d4bab27d6029bdb44e7fc7287633a446ea1bc189464af9fd03762"} Jul 28 12:27:22.149024 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:22.148987 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerStarted","Data":"3ab62da7a1b9ce586ee8d71434ea2b50be5fb89284d508ef1e6233444044c66a"} Jul 28 12:27:22.149522 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:22.149049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerStarted","Data":"2cb78463926925f543856ee37f4180db9fc25bc2ab5740fed4f1ce7f5ea46975"} Jul 28 12:27:22.149522 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:22.149068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerStarted","Data":"2c4de1bbf14e2f3e390bcef330457df51865fbb494339da4cec5128ff71e48eb"} Jul 28 12:27:22.149522 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:22.149086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerStarted","Data":"8adbde669d7f10ba0556ee8d6a803bf17bc16ffd71e76d1a2700c3a1dd8d6098"} Jul 28 12:27:22.178806 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:22.178735 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podStartSLOduration=14.066981097 podStartE2EDuration="27.178713397s" totalImagesPullingTime="13.1117323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:27:22.176631042 +0000 UTC m=+1455.993560860" watchObservedRunningTime="2026-07-28 12:27:22.178713397 +0000 UTC m=+1455.995643217" Jul 28 12:27:23.645640 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.645603 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:23.646128 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.645661 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:23.646128 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.645683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:23.646128 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.645705 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:23.647742 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.647698 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 28 12:27:23.647742 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.647724 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Jul 28 12:27:23.647978 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.647745 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Jul 28 12:27:23.647978 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:23.647820 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 12:27:26.646444 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.646385 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Jul 28 12:27:26.647179 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.646435 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 12:27:26.647179 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.646478 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Jul 28 12:27:26.647179 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.646493 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 28 12:27:26.647431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.646541 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:26.647431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.647262 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:26.647431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.647283 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:26.647431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:26.647300 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:29.646154 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:29.646126 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:29.646563 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:29.646393 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:29.646767 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:29.646751 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:29.646960 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:29.646943 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:29.651218 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:29.651203 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:30.190086 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:30.190055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:30.192822 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:30.192801 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:27:31.194859 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:27:31.194827 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:28:06.716429 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:06.716403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:28:06.719082 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:06.717390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:28:34.294980 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.294907 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn"] Jul 28 12:28:34.295372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.295183 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="registry" containerID="cri-o://8adbde669d7f10ba0556ee8d6a803bf17bc16ffd71e76d1a2700c3a1dd8d6098" gracePeriod=30 Jul 28 12:28:34.295372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.295235 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" containerID="cri-o://3ab62da7a1b9ce586ee8d71434ea2b50be5fb89284d508ef1e6233444044c66a" gracePeriod=30 Jul 28 12:28:34.295372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.295260 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="offline" containerID="cri-o://2cb78463926925f543856ee37f4180db9fc25bc2ab5740fed4f1ce7f5ea46975" gracePeriod=30 Jul 28 12:28:34.295372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.295254 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" containerID="cri-o://2c4de1bbf14e2f3e390bcef330457df51865fbb494339da4cec5128ff71e48eb" gracePeriod=30 Jul 28 12:28:34.659681 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.659595 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q"] Jul 28 12:28:34.664372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.664351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.667076 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.667054 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-hcb2d\"" Jul 28 12:28:34.678011 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.677988 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q"] Jul 28 12:28:34.735523 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.735498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.735636 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.735536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.735636 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.735551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2t9\" (UniqueName: \"kubernetes.io/projected/f422c72a-b574-4cfa-a918-c15806d9645d-kube-api-access-jj2t9\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.735636 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.735573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.735782 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.735655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f422c72a-b574-4cfa-a918-c15806d9645d-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.735782 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.735725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.837104 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.837068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.837104 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.837104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2t9\" (UniqueName: \"kubernetes.io/projected/f422c72a-b574-4cfa-a918-c15806d9645d-kube-api-access-jj2t9\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.837358 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.837128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.837358 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.837176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f422c72a-b574-4cfa-a918-c15806d9645d-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.837358 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.837221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.837358 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:28:34.837252 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 28 12:28:34.837358 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:28:34.837275 2564 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 28 12:28:34.837358 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.837306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.837358 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:28:34.837346 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls podName:f422c72a-b574-4cfa-a918-c15806d9645d nodeName:}" failed. No retries permitted until 2026-07-28 12:28:35.337321442 +0000 UTC m=+1529.154251243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls") pod "feast-simple-feast-setup-777d6cc998-qlt4q" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d") : secret "feast-simple-feast-setup-offline-tls" not found Jul 28 12:28:34.837678 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:28:34.837453 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls podName:f422c72a-b574-4cfa-a918-c15806d9645d nodeName:}" failed. No retries permitted until 2026-07-28 12:28:35.33742378 +0000 UTC m=+1529.154353574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls") pod "feast-simple-feast-setup-777d6cc998-qlt4q" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d") : secret "feast-simple-feast-setup-online-tls" not found Jul 28 12:28:34.837724 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.837702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f422c72a-b574-4cfa-a918-c15806d9645d-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.839955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.839927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.839955 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.839946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:34.859329 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:34.859305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2t9\" (UniqueName: \"kubernetes.io/projected/f422c72a-b574-4cfa-a918-c15806d9645d-kube-api-access-jj2t9\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:35.340705 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.340675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:35.340705 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.340708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:35.343017 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.343001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:35.343072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.343045 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-qlt4q\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:35.471056 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.471023 2564 generic.go:356] "Generic (PLEG): container finished" podID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerID="3ab62da7a1b9ce586ee8d71434ea2b50be5fb89284d508ef1e6233444044c66a" exitCode=0 Jul 28 12:28:35.471056 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.471048 2564 generic.go:356] "Generic (PLEG): container finished" podID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerID="2c4de1bbf14e2f3e390bcef330457df51865fbb494339da4cec5128ff71e48eb" exitCode=0 Jul 28 12:28:35.471223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.471083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerDied","Data":"3ab62da7a1b9ce586ee8d71434ea2b50be5fb89284d508ef1e6233444044c66a"} Jul 28 12:28:35.471223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.471122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerDied","Data":"2c4de1bbf14e2f3e390bcef330457df51865fbb494339da4cec5128ff71e48eb"} Jul 28 12:28:35.575051 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.575017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:35.709531 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:35.709504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q"] Jul 28 12:28:35.710992 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:28:35.710968 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf422c72a_b574_4cfa_a918_c15806d9645d.slice/crio-1607e9ad7d07b3a758688398b7ae8381b9c77d89354651f9a0b3b334c217d5ae WatchSource:0}: Error finding container 1607e9ad7d07b3a758688398b7ae8381b9c77d89354651f9a0b3b334c217d5ae: Status 404 returned error can't find the container with id 1607e9ad7d07b3a758688398b7ae8381b9c77d89354651f9a0b3b334c217d5ae Jul 28 12:28:36.476610 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:36.476576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerStarted","Data":"32cbab5b589fec861c7cb9fdd9159f3be7cc4c908215916d79150f74fcf28fc8"} Jul 28 12:28:36.476990 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:36.476619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerStarted","Data":"1607e9ad7d07b3a758688398b7ae8381b9c77d89354651f9a0b3b334c217d5ae"} Jul 28 12:28:39.647390 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:39.647347 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 28 12:28:40.190424 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:40.190388 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 12:28:40.493797 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:40.493768 2564 generic.go:356] "Generic (PLEG): container finished" podID="f422c72a-b574-4cfa-a918-c15806d9645d" containerID="32cbab5b589fec861c7cb9fdd9159f3be7cc4c908215916d79150f74fcf28fc8" exitCode=0 Jul 28 12:28:40.493959 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:40.493825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerDied","Data":"32cbab5b589fec861c7cb9fdd9159f3be7cc4c908215916d79150f74fcf28fc8"} Jul 28 12:28:41.500817 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:41.500783 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerStarted","Data":"e405b83d48ab455922ea8ff3c202709a8558487bd2791d74e8971f00a72bffe4"} Jul 28 12:28:46.522338 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:46.522307 2564 generic.go:356] "Generic (PLEG): container finished" podID="f422c72a-b574-4cfa-a918-c15806d9645d" containerID="e405b83d48ab455922ea8ff3c202709a8558487bd2791d74e8971f00a72bffe4" exitCode=0 Jul 28 12:28:46.522687 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:46.522383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerDied","Data":"e405b83d48ab455922ea8ff3c202709a8558487bd2791d74e8971f00a72bffe4"} Jul 28 12:28:47.533443 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.533406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerStarted","Data":"75e10159feac6aa014d034e82e836545720e84447c9e8fdce7b4de7ea1095d95"} Jul 28 12:28:47.533928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.533463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerStarted","Data":"8220eff81b2304ee8888e8d971986da8fc8208b45b0f283177b99a2bd4cd2e92"} Jul 28 12:28:47.533928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.533481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerStarted","Data":"00556b039439f3fd29af59503ab0fb86d1ea59e8c7d07b716ab3ff67edb4c454"} Jul 28 12:28:47.533928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.533498 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerStarted","Data":"6a72f4c5dd01cea0c2b14ce07a656c24526f6e69356a5295f745442623281932"} Jul 28 12:28:47.561334 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.561254 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podStartSLOduration=13.561233343 podStartE2EDuration="13.561233343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:28:47.558700596 +0000 UTC m=+1541.375630427" watchObservedRunningTime="2026-07-28 12:28:47.561233343 +0000 UTC m=+1541.378163161" Jul 28 12:28:47.576247 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.576215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:47.576404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.576272 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:47.576404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.576292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:47.576404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.576310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:47.577928 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.577895 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Jul 28 12:28:47.578064 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.577918 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 28 12:28:47.578216 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.578193 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Jul 28 12:28:47.578355 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:47.578260 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 28 12:28:49.647272 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:49.647228 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 28 12:28:50.190411 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:50.190371 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 12:28:50.575931 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:50.575861 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.32:8816: connect: connection refused" Jul 28 12:28:50.576132 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:50.575950 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused" Jul 28 12:28:50.576132 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:50.575892 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 28 12:28:50.576132 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:50.575895 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 28 12:28:53.576504 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.576476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.576870 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.576590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.576870 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.576812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.577179 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.577157 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.577284 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.577195 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.577284 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.577215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.577436 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.577415 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.577640 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.577627 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.577715 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.577685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:53.581413 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:53.581393 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:54.573632 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:54.573603 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:54.576832 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:54.576806 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:28:59.646737 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:59.646695 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.31:8443: connect: connection refused" Jul 28 12:28:59.647164 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:28:59.646814 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:29:00.189595 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:00.189546 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 12:29:00.189768 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:00.189716 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:29:04.625173 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:04.625102 2564 generic.go:356] "Generic (PLEG): container finished" podID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerID="2cb78463926925f543856ee37f4180db9fc25bc2ab5740fed4f1ce7f5ea46975" exitCode=137 Jul 28 12:29:04.625173 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:04.625131 2564 generic.go:356] "Generic (PLEG): container finished" podID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerID="8adbde669d7f10ba0556ee8d6a803bf17bc16ffd71e76d1a2700c3a1dd8d6098" exitCode=137 Jul 28 12:29:04.625173 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:04.625153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerDied","Data":"2cb78463926925f543856ee37f4180db9fc25bc2ab5740fed4f1ce7f5ea46975"} Jul 28 12:29:04.625559 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:04.625182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerDied","Data":"8adbde669d7f10ba0556ee8d6a803bf17bc16ffd71e76d1a2700c3a1dd8d6098"} Jul 28 12:29:04.941834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:04.941810 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:29:05.000262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.000240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzzzv\" (UniqueName: \"kubernetes.io/projected/419220c0-6ef6-4f01-a93b-3302a5d52207-kube-api-access-lzzzv\") pod \"419220c0-6ef6-4f01-a93b-3302a5d52207\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " Jul 28 12:29:05.000420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.000269 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-ui-tls\") pod \"419220c0-6ef6-4f01-a93b-3302a5d52207\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " Jul 28 12:29:05.000420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.000293 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls\") pod \"419220c0-6ef6-4f01-a93b-3302a5d52207\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " Jul 28 12:29:05.000420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.000335 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-online-tls\") pod \"419220c0-6ef6-4f01-a93b-3302a5d52207\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " Jul 28 12:29:05.000420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.000382 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/419220c0-6ef6-4f01-a93b-3302a5d52207-feast-data\") pod \"419220c0-6ef6-4f01-a93b-3302a5d52207\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " Jul 28 12:29:05.000420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.000422 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-offline-tls\") pod \"419220c0-6ef6-4f01-a93b-3302a5d52207\" (UID: \"419220c0-6ef6-4f01-a93b-3302a5d52207\") " Jul 28 12:29:05.001024 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.000986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419220c0-6ef6-4f01-a93b-3302a5d52207-feast-data" (OuterVolumeSpecName: "feast-data") pod "419220c0-6ef6-4f01-a93b-3302a5d52207" (UID: "419220c0-6ef6-4f01-a93b-3302a5d52207"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:29:05.001166 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.001069 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/419220c0-6ef6-4f01-a93b-3302a5d52207-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:29:05.002611 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.002588 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "419220c0-6ef6-4f01-a93b-3302a5d52207" (UID: "419220c0-6ef6-4f01-a93b-3302a5d52207"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:29:05.002746 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.002580 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419220c0-6ef6-4f01-a93b-3302a5d52207-kube-api-access-lzzzv" (OuterVolumeSpecName: "kube-api-access-lzzzv") pod "419220c0-6ef6-4f01-a93b-3302a5d52207" (UID: "419220c0-6ef6-4f01-a93b-3302a5d52207"). InnerVolumeSpecName "kube-api-access-lzzzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:29:05.002914 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.002870 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "419220c0-6ef6-4f01-a93b-3302a5d52207" (UID: "419220c0-6ef6-4f01-a93b-3302a5d52207"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:29:05.002976 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.002910 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-online-tls" (OuterVolumeSpecName: "online-tls") pod "419220c0-6ef6-4f01-a93b-3302a5d52207" (UID: "419220c0-6ef6-4f01-a93b-3302a5d52207"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:29:05.003092 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.003074 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "419220c0-6ef6-4f01-a93b-3302a5d52207" (UID: "419220c0-6ef6-4f01-a93b-3302a5d52207"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:29:05.101890 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.101865 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:29:05.101977 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.101905 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzzv\" (UniqueName: \"kubernetes.io/projected/419220c0-6ef6-4f01-a93b-3302a5d52207-kube-api-access-lzzzv\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:29:05.101977 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.101915 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:29:05.101977 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.101924 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:29:05.101977 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.101933 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/419220c0-6ef6-4f01-a93b-3302a5d52207-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:29:05.632361 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.632330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" event={"ID":"419220c0-6ef6-4f01-a93b-3302a5d52207","Type":"ContainerDied","Data":"362df739306e00477defd8db989894605a9f8f95fd3dbc108fe628d2be44067c"} Jul 28 12:29:05.632796 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.632380 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn" Jul 28 12:29:05.632796 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.632386 2564 scope.go:117] "RemoveContainer" containerID="3ab62da7a1b9ce586ee8d71434ea2b50be5fb89284d508ef1e6233444044c66a" Jul 28 12:29:05.643759 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.643733 2564 scope.go:117] "RemoveContainer" containerID="2cb78463926925f543856ee37f4180db9fc25bc2ab5740fed4f1ce7f5ea46975" Jul 28 12:29:05.652121 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.652105 2564 scope.go:117] "RemoveContainer" containerID="2c4de1bbf14e2f3e390bcef330457df51865fbb494339da4cec5128ff71e48eb" Jul 28 12:29:05.659838 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.659816 2564 scope.go:117] "RemoveContainer" containerID="8adbde669d7f10ba0556ee8d6a803bf17bc16ffd71e76d1a2700c3a1dd8d6098" Jul 28 12:29:05.660150 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.660106 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn"] Jul 28 12:29:05.663065 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.663046 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-7fvhn"] Jul 28 12:29:05.666926 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.666911 2564 scope.go:117] "RemoveContainer" containerID="dc438ccf941d4bab27d6029bdb44e7fc7287633a446ea1bc189464af9fd03762" Jul 28 12:29:05.673822 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:05.673806 2564 scope.go:117] "RemoveContainer" containerID="97bdfe48b878ffd29a25736f078eee3fca282ae798dc6eaa302abe7d56a4ffb1" Jul 28 12:29:06.779704 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:06.779671 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" path="/var/lib/kubelet/pods/419220c0-6ef6-4f01-a93b-3302a5d52207/volumes" Jul 28 12:29:56.465912 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.465866 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm"] Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466200 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466210 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466226 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466231 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466239 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="feast-init" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466246 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="feast-init" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466254 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="offline" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466259 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="offline" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466269 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="feast-apply" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466274 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="feast-apply" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466281 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="registry" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466285 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="registry" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466344 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="registry" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466352 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="ui" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466360 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="offline" Jul 28 12:29:56.466360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.466365 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="419220c0-6ef6-4f01-a93b-3302a5d52207" containerName="online" Jul 28 12:29:56.469991 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.469973 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.472542 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.472521 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 28 12:29:56.472645 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.472553 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 28 12:29:56.472645 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.472622 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 28 12:29:56.472645 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.472633 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 28 12:29:56.472809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.472665 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 28 12:29:56.472809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.472734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 28 12:29:56.472809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.472798 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-7wqzz\"" Jul 28 12:29:56.482535 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.482516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm"] Jul 28 12:29:56.500866 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.500843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.500958 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.500872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.500958 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.500943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5500d371-85a5-455a-8224-f4428bf4aa10-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.501033 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.500977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5500d371-85a5-455a-8224-f4428bf4aa10-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.501033 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.501006 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.501100 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.501036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9pm7\" (UniqueName: \"kubernetes.io/projected/5500d371-85a5-455a-8224-f4428bf4aa10-kube-api-access-x9pm7\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.601377 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.601357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5500d371-85a5-455a-8224-f4428bf4aa10-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.601487 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.601391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5500d371-85a5-455a-8224-f4428bf4aa10-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.601487 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.601415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.601487 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.601450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9pm7\" (UniqueName: \"kubernetes.io/projected/5500d371-85a5-455a-8224-f4428bf4aa10-kube-api-access-x9pm7\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.601487 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.601480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.601696 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.601507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.601696 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:29:56.601608 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 12:29:56.601696 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:29:56.601696 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls podName:5500d371-85a5-455a-8224-f4428bf4aa10 nodeName:}" failed. No retries permitted until 2026-07-28 12:29:57.101677437 +0000 UTC m=+1610.918607237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls") pod "feast-simple-feast-remote-setup-db6955c6c-q2wjm" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 12:29:56.601865 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.601781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5500d371-85a5-455a-8224-f4428bf4aa10-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.602112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.602093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5500d371-85a5-455a-8224-f4428bf4aa10-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.603963 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.603943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.604219 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.604200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:56.610428 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:56.610402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9pm7\" (UniqueName: \"kubernetes.io/projected/5500d371-85a5-455a-8224-f4428bf4aa10-kube-api-access-x9pm7\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:57.109944 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:57.106279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:57.109944 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:29:57.106477 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 12:29:57.109944 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:29:57.106590 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls podName:5500d371-85a5-455a-8224-f4428bf4aa10 nodeName:}" failed. No retries permitted until 2026-07-28 12:29:58.10655873 +0000 UTC m=+1611.923488540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls") pod "feast-simple-feast-remote-setup-db6955c6c-q2wjm" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 12:29:58.114360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:58.114330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:58.116698 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:58.116675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-q2wjm\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:58.278966 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:58.278945 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:29:58.415802 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:58.415774 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm"] Jul 28 12:29:58.417542 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:29:58.417510 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5500d371_85a5_455a_8224_f4428bf4aa10.slice/crio-d72c80285fff13bd2af22b65c382c73bc4db9d3b942458b89a12ef54e6584079 WatchSource:0}: Error finding container d72c80285fff13bd2af22b65c382c73bc4db9d3b942458b89a12ef54e6584079: Status 404 returned error can't find the container with id d72c80285fff13bd2af22b65c382c73bc4db9d3b942458b89a12ef54e6584079 Jul 28 12:29:58.860008 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:58.859972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerStarted","Data":"5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba"} Jul 28 12:29:58.860008 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:29:58.860013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerStarted","Data":"d72c80285fff13bd2af22b65c382c73bc4db9d3b942458b89a12ef54e6584079"} Jul 28 12:30:03.882227 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:03.882195 2564 generic.go:356] "Generic (PLEG): container finished" podID="5500d371-85a5-455a-8224-f4428bf4aa10" containerID="5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba" exitCode=0 Jul 28 12:30:03.882586 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:03.882267 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerDied","Data":"5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba"} Jul 28 12:30:04.888319 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:04.888284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerStarted","Data":"06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9"} Jul 28 12:30:09.910323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:09.910247 2564 generic.go:356] "Generic (PLEG): container finished" podID="5500d371-85a5-455a-8224-f4428bf4aa10" containerID="06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9" exitCode=0 Jul 28 12:30:09.910323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:09.910286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerDied","Data":"06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9"} Jul 28 12:30:10.918122 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:10.918084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerStarted","Data":"dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a"} Jul 28 12:30:10.918593 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:10.918136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerStarted","Data":"558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b"} Jul 28 12:30:10.918593 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:10.918153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerStarted","Data":"bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d"} Jul 28 12:30:10.944632 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:10.944587 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podStartSLOduration=14.94457214 podStartE2EDuration="14.94457214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:10.942494811 +0000 UTC m=+1624.759424631" watchObservedRunningTime="2026-07-28 12:30:10.94457214 +0000 UTC m=+1624.761501956" Jul 28 12:30:13.279562 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:13.279502 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:13.279562 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:13.279555 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:13.280086 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:13.279575 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:13.281370 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:13.281338 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.33:8816: connect: connection refused" Jul 28 12:30:13.281370 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:13.281355 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 28 12:30:13.281541 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:13.281397 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 28 12:30:16.280907 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.280849 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.281359 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.281069 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.281554 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.281531 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.281702 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.281570 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.281782 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.281765 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.282009 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.281992 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.287518 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.287499 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.949323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.949287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:30:16.954061 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:30:16.954036 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:31:19.934207 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:19.934172 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm"] Jul 28 12:31:19.934640 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:19.934452 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" containerID="cri-o://bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d" gracePeriod=30 Jul 28 12:31:19.934640 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:19.934548 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" containerID="cri-o://dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a" gracePeriod=30 Jul 28 12:31:19.934640 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:19.934568 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="offline" containerID="cri-o://558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b" gracePeriod=30 Jul 28 12:31:20.097360 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:20.097320 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q"] Jul 28 12:31:20.097646 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:20.097550 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="registry" containerID="cri-o://6a72f4c5dd01cea0c2b14ce07a656c24526f6e69356a5295f745442623281932" gracePeriod=30 Jul 28 12:31:20.097646 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:20.097611 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="offline" containerID="cri-o://8220eff81b2304ee8888e8d971986da8fc8208b45b0f283177b99a2bd4cd2e92" gracePeriod=30 Jul 28 12:31:20.097646 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:20.097625 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" containerID="cri-o://75e10159feac6aa014d034e82e836545720e84447c9e8fdce7b4de7ea1095d95" gracePeriod=30 Jul 28 12:31:20.097918 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:20.097618 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" containerID="cri-o://00556b039439f3fd29af59503ab0fb86d1ea59e8c7d07b716ab3ff67edb4c454" gracePeriod=30 Jul 28 12:31:21.224404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.224379 2564 generic.go:356] "Generic (PLEG): container finished" podID="5500d371-85a5-455a-8224-f4428bf4aa10" containerID="dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a" exitCode=0 Jul 28 12:31:21.224404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.224402 2564 generic.go:356] "Generic (PLEG): container finished" podID="5500d371-85a5-455a-8224-f4428bf4aa10" containerID="bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d" exitCode=0 Jul 28 12:31:21.224829 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.224457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerDied","Data":"dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a"} Jul 28 12:31:21.224829 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.224508 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerDied","Data":"bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d"} Jul 28 12:31:21.227244 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.227225 2564 generic.go:356] "Generic (PLEG): container finished" podID="f422c72a-b574-4cfa-a918-c15806d9645d" containerID="75e10159feac6aa014d034e82e836545720e84447c9e8fdce7b4de7ea1095d95" exitCode=0 Jul 28 12:31:21.227244 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.227240 2564 generic.go:356] "Generic (PLEG): container finished" podID="f422c72a-b574-4cfa-a918-c15806d9645d" containerID="00556b039439f3fd29af59503ab0fb86d1ea59e8c7d07b716ab3ff67edb4c454" exitCode=0 Jul 28 12:31:21.227379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.227278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerDied","Data":"75e10159feac6aa014d034e82e836545720e84447c9e8fdce7b4de7ea1095d95"} Jul 28 12:31:21.227379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:21.227302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerDied","Data":"00556b039439f3fd29af59503ab0fb86d1ea59e8c7d07b716ab3ff67edb4c454"} Jul 28 12:31:23.577931 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:23.577847 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 28 12:31:24.574283 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:24.574234 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 28 12:31:26.282391 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:26.282352 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 28 12:31:26.950382 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:26.950335 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 28 12:31:33.577764 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:33.577721 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 28 12:31:34.574562 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:34.574520 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 28 12:31:36.282301 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:36.282251 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 28 12:31:36.950363 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:36.950323 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 28 12:31:43.577665 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:43.577621 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.32:8443: connect: connection refused" Jul 28 12:31:43.578123 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:43.577792 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:31:44.573658 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:44.573617 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused" Jul 28 12:31:44.573846 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:44.573748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:31:46.282272 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:46.282228 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.33:8443: connect: connection refused" Jul 28 12:31:46.282709 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:46.282363 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:31:46.950072 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:46.950034 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 28 12:31:46.950375 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:46.950137 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:31:50.125794 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.125768 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:31:50.221829 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.221804 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls\") pod \"5500d371-85a5-455a-8224-f4428bf4aa10\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " Jul 28 12:31:50.222000 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.221848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5500d371-85a5-455a-8224-f4428bf4aa10-registry-tls\") pod \"5500d371-85a5-455a-8224-f4428bf4aa10\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " Jul 28 12:31:50.222000 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.221929 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5500d371-85a5-455a-8224-f4428bf4aa10-feast-data\") pod \"5500d371-85a5-455a-8224-f4428bf4aa10\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " Jul 28 12:31:50.222000 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.221989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-online-tls\") pod \"5500d371-85a5-455a-8224-f4428bf4aa10\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " Jul 28 12:31:50.222173 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.222017 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9pm7\" (UniqueName: \"kubernetes.io/projected/5500d371-85a5-455a-8224-f4428bf4aa10-kube-api-access-x9pm7\") pod \"5500d371-85a5-455a-8224-f4428bf4aa10\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " Jul 28 12:31:50.222173 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.222041 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-offline-tls\") pod \"5500d371-85a5-455a-8224-f4428bf4aa10\" (UID: \"5500d371-85a5-455a-8224-f4428bf4aa10\") " Jul 28 12:31:50.222359 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.222325 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5500d371-85a5-455a-8224-f4428bf4aa10-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5500d371-85a5-455a-8224-f4428bf4aa10" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:31:50.222550 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.222526 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5500d371-85a5-455a-8224-f4428bf4aa10-feast-data" (OuterVolumeSpecName: "feast-data") pod "5500d371-85a5-455a-8224-f4428bf4aa10" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:31:50.224222 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.224193 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5500d371-85a5-455a-8224-f4428bf4aa10" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:31:50.224325 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.224239 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-online-tls" (OuterVolumeSpecName: "online-tls") pod "5500d371-85a5-455a-8224-f4428bf4aa10" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:31:50.224325 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.224243 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5500d371-85a5-455a-8224-f4428bf4aa10-kube-api-access-x9pm7" (OuterVolumeSpecName: "kube-api-access-x9pm7") pod "5500d371-85a5-455a-8224-f4428bf4aa10" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10"). InnerVolumeSpecName "kube-api-access-x9pm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:31:50.224465 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.224436 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5500d371-85a5-455a-8224-f4428bf4aa10" (UID: "5500d371-85a5-455a-8224-f4428bf4aa10"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:31:50.322897 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.322855 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.322897 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.322893 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5500d371-85a5-455a-8224-f4428bf4aa10-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.323039 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.322903 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5500d371-85a5-455a-8224-f4428bf4aa10-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.323039 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.322910 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.323039 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.322918 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9pm7\" (UniqueName: \"kubernetes.io/projected/5500d371-85a5-455a-8224-f4428bf4aa10-kube-api-access-x9pm7\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.323039 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.322927 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5500d371-85a5-455a-8224-f4428bf4aa10-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.354108 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.354086 2564 generic.go:356] "Generic (PLEG): container finished" podID="f422c72a-b574-4cfa-a918-c15806d9645d" containerID="8220eff81b2304ee8888e8d971986da8fc8208b45b0f283177b99a2bd4cd2e92" exitCode=137 Jul 28 12:31:50.354108 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.354106 2564 generic.go:356] "Generic (PLEG): container finished" podID="f422c72a-b574-4cfa-a918-c15806d9645d" containerID="6a72f4c5dd01cea0c2b14ce07a656c24526f6e69356a5295f745442623281932" exitCode=137 Jul 28 12:31:50.354224 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.354158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerDied","Data":"8220eff81b2304ee8888e8d971986da8fc8208b45b0f283177b99a2bd4cd2e92"} Jul 28 12:31:50.354224 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.354203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerDied","Data":"6a72f4c5dd01cea0c2b14ce07a656c24526f6e69356a5295f745442623281932"} Jul 28 12:31:50.356534 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.356513 2564 generic.go:356] "Generic (PLEG): container finished" podID="5500d371-85a5-455a-8224-f4428bf4aa10" containerID="558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b" exitCode=137 Jul 28 12:31:50.356615 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.356571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerDied","Data":"558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b"} Jul 28 12:31:50.356615 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.356604 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" Jul 28 12:31:50.356690 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.356620 2564 scope.go:117] "RemoveContainer" containerID="dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a" Jul 28 12:31:50.356737 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.356607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm" event={"ID":"5500d371-85a5-455a-8224-f4428bf4aa10","Type":"ContainerDied","Data":"d72c80285fff13bd2af22b65c382c73bc4db9d3b942458b89a12ef54e6584079"} Jul 28 12:31:50.365619 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.365604 2564 scope.go:117] "RemoveContainer" containerID="558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b" Jul 28 12:31:50.372755 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.372738 2564 scope.go:117] "RemoveContainer" containerID="bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d" Jul 28 12:31:50.380160 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.380146 2564 scope.go:117] "RemoveContainer" containerID="06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9" Jul 28 12:31:50.383163 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.383143 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm"] Jul 28 12:31:50.387408 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.387389 2564 scope.go:117] "RemoveContainer" containerID="5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba" Jul 28 12:31:50.392094 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.392074 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-q2wjm"] Jul 28 12:31:50.395124 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.395110 2564 scope.go:117] "RemoveContainer" containerID="dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a" Jul 28 12:31:50.395360 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:31:50.395345 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a\": container with ID starting with dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a not found: ID does not exist" containerID="dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a" Jul 28 12:31:50.395412 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.395375 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a"} err="failed to get container status \"dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a\": rpc error: code = NotFound desc = could not find container \"dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a\": container with ID starting with dd40ad0ec8ca9bcd46c3d3491671c1e8e912246591642144a229fe89e7bc2c6a not found: ID does not exist" Jul 28 12:31:50.395412 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.395390 2564 scope.go:117] "RemoveContainer" containerID="558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b" Jul 28 12:31:50.395626 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:31:50.395606 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b\": container with ID starting with 558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b not found: ID does not exist" containerID="558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b" Jul 28 12:31:50.395693 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.395634 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b"} err="failed to get container status \"558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b\": rpc error: code = NotFound desc = could not find container \"558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b\": container with ID starting with 558abf694f7ff7399ed05b15475b0f3fab0779d8ef90376411b43f0588a2f19b not found: ID does not exist" Jul 28 12:31:50.395693 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.395667 2564 scope.go:117] "RemoveContainer" containerID="bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d" Jul 28 12:31:50.395978 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:31:50.395957 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d\": container with ID starting with bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d not found: ID does not exist" containerID="bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d" Jul 28 12:31:50.396026 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.395989 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d"} err="failed to get container status \"bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d\": rpc error: code = NotFound desc = could not find container \"bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d\": container with ID starting with bd13996e6e07fbff127d798f8ecbe3572373706e157a66f05380a4428cc6776d not found: ID does not exist" Jul 28 12:31:50.396026 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.396008 2564 scope.go:117] "RemoveContainer" containerID="06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9" Jul 28 12:31:50.396240 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:31:50.396224 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9\": container with ID starting with 06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9 not found: ID does not exist" containerID="06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9" Jul 28 12:31:50.396284 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.396245 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9"} err="failed to get container status \"06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9\": rpc error: code = NotFound desc = could not find container \"06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9\": container with ID starting with 06c121bbee10ffffa8b5ed842fe85b426a6ba0b61ca4ac3a07a5299a80c61ad9 not found: ID does not exist" Jul 28 12:31:50.396284 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.396261 2564 scope.go:117] "RemoveContainer" containerID="5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba" Jul 28 12:31:50.396486 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:31:50.396470 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba\": container with ID starting with 5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba not found: ID does not exist" containerID="5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba" Jul 28 12:31:50.396533 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.396490 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba"} err="failed to get container status \"5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba\": rpc error: code = NotFound desc = could not find container \"5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba\": container with ID starting with 5152aac4f1cfc5714e3e1748939decb6cf9b6fe0399b61728d23b45ec165a8ba not found: ID does not exist" Jul 28 12:31:50.744785 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.744765 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:31:50.780018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.779984 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" path="/var/lib/kubelet/pods/5500d371-85a5-455a-8224-f4428bf4aa10/volumes" Jul 28 12:31:50.825647 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.825625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls\") pod \"f422c72a-b574-4cfa-a918-c15806d9645d\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " Jul 28 12:31:50.825755 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.825672 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f422c72a-b574-4cfa-a918-c15806d9645d-feast-data\") pod \"f422c72a-b574-4cfa-a918-c15806d9645d\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " Jul 28 12:31:50.825755 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.825710 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj2t9\" (UniqueName: \"kubernetes.io/projected/f422c72a-b574-4cfa-a918-c15806d9645d-kube-api-access-jj2t9\") pod \"f422c72a-b574-4cfa-a918-c15806d9645d\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " Jul 28 12:31:50.825904 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.825775 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-registry-tls\") pod \"f422c72a-b574-4cfa-a918-c15806d9645d\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " Jul 28 12:31:50.825904 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.825801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-ui-tls\") pod \"f422c72a-b574-4cfa-a918-c15806d9645d\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " Jul 28 12:31:50.825904 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.825842 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls\") pod \"f422c72a-b574-4cfa-a918-c15806d9645d\" (UID: \"f422c72a-b574-4cfa-a918-c15806d9645d\") " Jul 28 12:31:50.826302 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.826275 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f422c72a-b574-4cfa-a918-c15806d9645d-feast-data" (OuterVolumeSpecName: "feast-data") pod "f422c72a-b574-4cfa-a918-c15806d9645d" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:31:50.828116 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.828089 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f422c72a-b574-4cfa-a918-c15806d9645d" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:31:50.828230 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.828119 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f422c72a-b574-4cfa-a918-c15806d9645d" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:31:50.828230 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.828138 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls" (OuterVolumeSpecName: "online-tls") pod "f422c72a-b574-4cfa-a918-c15806d9645d" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:31:50.828431 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.828414 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f422c72a-b574-4cfa-a918-c15806d9645d" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:31:50.828479 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.828433 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f422c72a-b574-4cfa-a918-c15806d9645d-kube-api-access-jj2t9" (OuterVolumeSpecName: "kube-api-access-jj2t9") pod "f422c72a-b574-4cfa-a918-c15806d9645d" (UID: "f422c72a-b574-4cfa-a918-c15806d9645d"). InnerVolumeSpecName "kube-api-access-jj2t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:31:50.926372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.926323 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.926372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.926341 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f422c72a-b574-4cfa-a918-c15806d9645d-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.926372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.926351 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2t9\" (UniqueName: \"kubernetes.io/projected/f422c72a-b574-4cfa-a918-c15806d9645d-kube-api-access-jj2t9\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.926372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.926359 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.926372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.926367 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:50.926372 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:50.926375 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f422c72a-b574-4cfa-a918-c15806d9645d-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:31:51.363323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.363304 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" Jul 28 12:31:51.363323 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.363310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q" event={"ID":"f422c72a-b574-4cfa-a918-c15806d9645d","Type":"ContainerDied","Data":"1607e9ad7d07b3a758688398b7ae8381b9c77d89354651f9a0b3b334c217d5ae"} Jul 28 12:31:51.363765 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.363361 2564 scope.go:117] "RemoveContainer" containerID="75e10159feac6aa014d034e82e836545720e84447c9e8fdce7b4de7ea1095d95" Jul 28 12:31:51.372111 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.372095 2564 scope.go:117] "RemoveContainer" containerID="8220eff81b2304ee8888e8d971986da8fc8208b45b0f283177b99a2bd4cd2e92" Jul 28 12:31:51.380003 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.379987 2564 scope.go:117] "RemoveContainer" containerID="00556b039439f3fd29af59503ab0fb86d1ea59e8c7d07b716ab3ff67edb4c454" Jul 28 12:31:51.385339 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.385320 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q"] Jul 28 12:31:51.387548 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.387533 2564 scope.go:117] "RemoveContainer" containerID="6a72f4c5dd01cea0c2b14ce07a656c24526f6e69356a5295f745442623281932" Jul 28 12:31:51.390919 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.390902 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-qlt4q"] Jul 28 12:31:51.395099 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.395082 2564 scope.go:117] "RemoveContainer" containerID="e405b83d48ab455922ea8ff3c202709a8558487bd2791d74e8971f00a72bffe4" Jul 28 12:31:51.402006 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:51.401990 2564 scope.go:117] "RemoveContainer" containerID="32cbab5b589fec861c7cb9fdd9159f3be7cc4c908215916d79150f74fcf28fc8" Jul 28 12:31:52.780503 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:31:52.780469 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" path="/var/lib/kubelet/pods/f422c72a-b574-4cfa-a918-c15806d9645d/volumes" Jul 28 12:32:33.327518 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327486 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj"] Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327799 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="feast-init" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327809 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="feast-init" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327819 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327825 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327835 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="offline" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327840 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="offline" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327849 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="feast-init" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327857 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="feast-init" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327864 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="registry" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327872 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="registry" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327898 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327905 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327914 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="feast-apply" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327922 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="feast-apply" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327932 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="offline" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327940 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="offline" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327949 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327957 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327978 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="feast-apply" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.327987 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="feast-apply" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328006 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" Jul 28 12:32:33.328015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328015 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" Jul 28 12:32:33.328706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328105 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="registry" Jul 28 12:32:33.328706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328118 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="offline" Jul 28 12:32:33.328706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328128 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="online" Jul 28 12:32:33.328706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328136 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="ui" Jul 28 12:32:33.328706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328145 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5500d371-85a5-455a-8224-f4428bf4aa10" containerName="offline" Jul 28 12:32:33.328706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328156 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="ui" Jul 28 12:32:33.328706 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.328163 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f422c72a-b574-4cfa-a918-c15806d9645d" containerName="online" Jul 28 12:32:33.333343 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.333327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.336228 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.336208 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-6n2c4\"" Jul 28 12:32:33.336388 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.336371 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 12:32:33.336463 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.336407 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 28 12:32:33.336524 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.336456 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 28 12:32:33.337131 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.337112 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 28 12:32:33.337131 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.337120 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 12:32:33.337317 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.337237 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 28 12:32:33.349674 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.349652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj"] Jul 28 12:32:33.437322 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.437300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.437432 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.437336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.437432 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.437357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe840242-fdf5-4733-8e08-c876f87401f2-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.437432 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.437399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.437538 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.437432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.437538 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.437449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjgtg\" (UniqueName: \"kubernetes.io/projected/fe840242-fdf5-4733-8e08-c876f87401f2-kube-api-access-cjgtg\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.538566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.538540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.538686 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.538588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.538686 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.538609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe840242-fdf5-4733-8e08-c876f87401f2-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.538686 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.538634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.538686 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.538654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.538915 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.538697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjgtg\" (UniqueName: \"kubernetes.io/projected/fe840242-fdf5-4733-8e08-c876f87401f2-kube-api-access-cjgtg\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.539120 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.539102 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe840242-fdf5-4733-8e08-c876f87401f2-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.541363 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.541320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.541363 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.541320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.541544 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.541526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.541745 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.541603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.550399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.550381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjgtg\" (UniqueName: \"kubernetes.io/projected/fe840242-fdf5-4733-8e08-c876f87401f2-kube-api-access-cjgtg\") pod \"feast-simple-feast-setup-777d6cc998-s75rj\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.654983 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.654908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:33.805608 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:33.805587 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj"] Jul 28 12:32:33.807432 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:32:33.807410 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe840242_fdf5_4733_8e08_c876f87401f2.slice/crio-0d6bc3b66a499c1b0adc1c730e5716a950e8053337dc8ebdf8365349a92fdedb WatchSource:0}: Error finding container 0d6bc3b66a499c1b0adc1c730e5716a950e8053337dc8ebdf8365349a92fdedb: Status 404 returned error can't find the container with id 0d6bc3b66a499c1b0adc1c730e5716a950e8053337dc8ebdf8365349a92fdedb Jul 28 12:32:34.542573 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:34.542539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerStarted","Data":"4cee7078d83ddad0068d17bcb4b49bac54a23661dc4405402a2c14d7e69f9507"} Jul 28 12:32:34.542573 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:34.542578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerStarted","Data":"0d6bc3b66a499c1b0adc1c730e5716a950e8053337dc8ebdf8365349a92fdedb"} Jul 28 12:32:38.561377 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:38.561348 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe840242-fdf5-4733-8e08-c876f87401f2" containerID="4cee7078d83ddad0068d17bcb4b49bac54a23661dc4405402a2c14d7e69f9507" exitCode=0 Jul 28 12:32:38.561798 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:38.561419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerDied","Data":"4cee7078d83ddad0068d17bcb4b49bac54a23661dc4405402a2c14d7e69f9507"} Jul 28 12:32:39.566914 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:39.566857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerStarted","Data":"677e7acb7a3f79bdac93731583993ab767973be084987a382c46aaabba355d6e"} Jul 28 12:32:44.588702 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:44.588669 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe840242-fdf5-4733-8e08-c876f87401f2" containerID="677e7acb7a3f79bdac93731583993ab767973be084987a382c46aaabba355d6e" exitCode=0 Jul 28 12:32:44.589100 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:44.588746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerDied","Data":"677e7acb7a3f79bdac93731583993ab767973be084987a382c46aaabba355d6e"} Jul 28 12:32:45.598657 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.598619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerStarted","Data":"ed5d674ad6f67d48125830ed57e476e1cd83b762ec7f2c401815dc421cfd24b4"} Jul 28 12:32:45.599163 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.598693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerStarted","Data":"bc2e15d41d02a63df9e3d12370e71bcaa96367aaf3ab6634fc7280a6bcd1c364"} Jul 28 12:32:45.599163 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.598715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerStarted","Data":"c56022d473a450085fdb7654a6e0fdfb728584aca8a4194d82c9878e13a79731"} Jul 28 12:32:45.599163 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.598734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerStarted","Data":"990354fbcf06585173964e33631bdd80ccc35ec8bd2660535494d2876c29f26f"} Jul 28 12:32:45.629153 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.629098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podStartSLOduration=12.629075139 podStartE2EDuration="12.629075139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:32:45.627481583 +0000 UTC m=+1779.444411402" watchObservedRunningTime="2026-07-28 12:32:45.629075139 +0000 UTC m=+1779.446004957" Jul 28 12:32:45.655833 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.655796 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:45.656107 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.656089 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:45.656245 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.656231 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:45.656356 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.656344 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:45.658059 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.658020 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.34:8816: connect: connection refused" Jul 28 12:32:45.658059 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.658045 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused" Jul 28 12:32:45.658237 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.658058 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 28 12:32:45.658237 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:45.658144 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 12:32:48.655478 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:48.655434 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused" Jul 28 12:32:48.656099 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:48.655446 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.34:8816: connect: connection refused" Jul 28 12:32:48.656240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:48.655448 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 28 12:32:48.656240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:48.655469 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 12:32:51.655981 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.655950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.656438 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.656262 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.656438 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.656347 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.656729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.656703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.656903 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.656748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.656903 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.656768 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.657030 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.657014 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.657099 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.657077 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.657200 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.657141 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:51.661181 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:51.661165 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:52.634801 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:52.634776 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:32:52.637927 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:32:52.637905 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:33:06.750749 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:06.750649 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:33:06.763499 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:06.752757 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:33:54.873379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:54.873345 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj"] Jul 28 12:33:54.873993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:54.873586 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="registry" containerID="cri-o://990354fbcf06585173964e33631bdd80ccc35ec8bd2660535494d2876c29f26f" gracePeriod=30 Jul 28 12:33:54.873993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:54.873609 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="offline" containerID="cri-o://bc2e15d41d02a63df9e3d12370e71bcaa96367aaf3ab6634fc7280a6bcd1c364" gracePeriod=30 Jul 28 12:33:54.873993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:54.873650 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" containerID="cri-o://c56022d473a450085fdb7654a6e0fdfb728584aca8a4194d82c9878e13a79731" gracePeriod=30 Jul 28 12:33:54.873993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:54.873650 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" containerID="cri-o://ed5d674ad6f67d48125830ed57e476e1cd83b762ec7f2c401815dc421cfd24b4" gracePeriod=30 Jul 28 12:33:55.289606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.289564 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms"] Jul 28 12:33:55.293558 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.293538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.296152 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.296118 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-g9r7n\"" Jul 28 12:33:55.308084 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.308061 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms"] Jul 28 12:33:55.384293 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.384257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b098a990-1f03-4fe9-8389-769ee5f74283-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.384427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.384310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlthh\" (UniqueName: \"kubernetes.io/projected/b098a990-1f03-4fe9-8389-769ee5f74283-kube-api-access-qlthh\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.384427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.384356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.384427 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.384416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.384533 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.384445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.384533 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.384470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.485341 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.485302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.485341 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.485340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.485592 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.485362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.485592 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.485402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b098a990-1f03-4fe9-8389-769ee5f74283-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.485592 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.485425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlthh\" (UniqueName: \"kubernetes.io/projected/b098a990-1f03-4fe9-8389-769ee5f74283-kube-api-access-qlthh\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.485592 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.485459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.485953 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.485930 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b098a990-1f03-4fe9-8389-769ee5f74283-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.488155 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.488132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.488420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.488403 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.488459 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.488425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.488515 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.488494 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.495802 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.495779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlthh\" (UniqueName: \"kubernetes.io/projected/b098a990-1f03-4fe9-8389-769ee5f74283-kube-api-access-qlthh\") pod \"feast-simple-feast-setup-777d6cc998-h58ms\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.603351 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.603330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:33:55.744222 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.744192 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms"] Jul 28 12:33:55.747627 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:33:55.747589 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb098a990_1f03_4fe9_8389_769ee5f74283.slice/crio-2c285d7378bf0a03f89b74559306d98407af64b393a5a3db60be8f1971fb462e WatchSource:0}: Error finding container 2c285d7378bf0a03f89b74559306d98407af64b393a5a3db60be8f1971fb462e: Status 404 returned error can't find the container with id 2c285d7378bf0a03f89b74559306d98407af64b393a5a3db60be8f1971fb462e Jul 28 12:33:55.896190 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.896120 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe840242-fdf5-4733-8e08-c876f87401f2" containerID="ed5d674ad6f67d48125830ed57e476e1cd83b762ec7f2c401815dc421cfd24b4" exitCode=0 Jul 28 12:33:55.896190 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.896143 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe840242-fdf5-4733-8e08-c876f87401f2" containerID="c56022d473a450085fdb7654a6e0fdfb728584aca8a4194d82c9878e13a79731" exitCode=0 Jul 28 12:33:55.896190 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.896148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerDied","Data":"ed5d674ad6f67d48125830ed57e476e1cd83b762ec7f2c401815dc421cfd24b4"} Jul 28 12:33:55.896190 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.896186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerDied","Data":"c56022d473a450085fdb7654a6e0fdfb728584aca8a4194d82c9878e13a79731"} Jul 28 12:33:55.897725 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.897699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerStarted","Data":"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953"} Jul 28 12:33:55.897832 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:33:55.897743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerStarted","Data":"2c285d7378bf0a03f89b74559306d98407af64b393a5a3db60be8f1971fb462e"} Jul 28 12:34:00.919950 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:00.919918 2564 generic.go:356] "Generic (PLEG): container finished" podID="b098a990-1f03-4fe9-8389-769ee5f74283" containerID="a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953" exitCode=0 Jul 28 12:34:00.920307 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:00.919998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerDied","Data":"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953"} Jul 28 12:34:01.656849 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:01.656803 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused" Jul 28 12:34:01.927080 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:01.927001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerStarted","Data":"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20"} Jul 28 12:34:02.634896 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:02.634844 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 12:34:06.950553 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:06.950523 2564 generic.go:356] "Generic (PLEG): container finished" podID="b098a990-1f03-4fe9-8389-769ee5f74283" containerID="635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20" exitCode=0 Jul 28 12:34:06.951015 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:06.950593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerDied","Data":"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20"} Jul 28 12:34:07.961547 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:07.961502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerStarted","Data":"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec"} Jul 28 12:34:07.962037 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:07.961571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerStarted","Data":"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368"} Jul 28 12:34:07.962037 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:07.961591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerStarted","Data":"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd"} Jul 28 12:34:07.962037 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:07.961608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerStarted","Data":"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58"} Jul 28 12:34:07.988648 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:07.988572 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podStartSLOduration=12.988550607 podStartE2EDuration="12.988550607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:34:07.988427858 +0000 UTC m=+1861.805357677" watchObservedRunningTime="2026-07-28 12:34:07.988550607 +0000 UTC m=+1861.805480426" Jul 28 12:34:10.603697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.603663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:10.604298 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.604270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:10.604437 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.604307 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:10.604437 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.604325 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:10.605590 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.605554 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.35:8816: connect: connection refused" Jul 28 12:34:10.605710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.605671 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 12:34:10.605783 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.605749 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Jul 28 12:34:10.605858 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:10.605839 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Jul 28 12:34:11.657022 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:11.656972 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused" Jul 28 12:34:12.635601 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:12.635536 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 12:34:13.604709 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.604679 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605181 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.604779 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605181 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.605004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605333 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.605317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605392 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.605355 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605392 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.605374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605739 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.605715 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605849 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.605784 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.605849 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.605830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.609957 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.609939 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:13.997195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:13.997169 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:14.000507 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:14.000486 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:34:21.657271 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:21.657228 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.34:8443: connect: connection refused" Jul 28 12:34:21.657746 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:21.657372 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:34:22.635514 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:22.635475 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 12:34:22.635690 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:22.635593 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:34:25.045793 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.045758 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe840242-fdf5-4733-8e08-c876f87401f2" containerID="bc2e15d41d02a63df9e3d12370e71bcaa96367aaf3ab6634fc7280a6bcd1c364" exitCode=137 Jul 28 12:34:25.045793 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.045783 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe840242-fdf5-4733-8e08-c876f87401f2" containerID="990354fbcf06585173964e33631bdd80ccc35ec8bd2660535494d2876c29f26f" exitCode=137 Jul 28 12:34:25.046228 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.045829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerDied","Data":"bc2e15d41d02a63df9e3d12370e71bcaa96367aaf3ab6634fc7280a6bcd1c364"} Jul 28 12:34:25.046228 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.045895 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerDied","Data":"990354fbcf06585173964e33631bdd80ccc35ec8bd2660535494d2876c29f26f"} Jul 28 12:34:25.530206 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.530184 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:34:25.636979 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.636946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-ui-tls\") pod \"fe840242-fdf5-4733-8e08-c876f87401f2\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " Jul 28 12:34:25.637125 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.637001 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-registry-tls\") pod \"fe840242-fdf5-4733-8e08-c876f87401f2\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " Jul 28 12:34:25.637125 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.637021 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjgtg\" (UniqueName: \"kubernetes.io/projected/fe840242-fdf5-4733-8e08-c876f87401f2-kube-api-access-cjgtg\") pod \"fe840242-fdf5-4733-8e08-c876f87401f2\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " Jul 28 12:34:25.637125 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.637051 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-online-tls\") pod \"fe840242-fdf5-4733-8e08-c876f87401f2\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " Jul 28 12:34:25.637125 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.637087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe840242-fdf5-4733-8e08-c876f87401f2-feast-data\") pod \"fe840242-fdf5-4733-8e08-c876f87401f2\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " Jul 28 12:34:25.637363 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.637131 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-offline-tls\") pod \"fe840242-fdf5-4733-8e08-c876f87401f2\" (UID: \"fe840242-fdf5-4733-8e08-c876f87401f2\") " Jul 28 12:34:25.637653 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.637593 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe840242-fdf5-4733-8e08-c876f87401f2-feast-data" (OuterVolumeSpecName: "feast-data") pod "fe840242-fdf5-4733-8e08-c876f87401f2" (UID: "fe840242-fdf5-4733-8e08-c876f87401f2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:34:25.639399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.639366 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe840242-fdf5-4733-8e08-c876f87401f2-kube-api-access-cjgtg" (OuterVolumeSpecName: "kube-api-access-cjgtg") pod "fe840242-fdf5-4733-8e08-c876f87401f2" (UID: "fe840242-fdf5-4733-8e08-c876f87401f2"). InnerVolumeSpecName "kube-api-access-cjgtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:34:25.639512 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.639402 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-online-tls" (OuterVolumeSpecName: "online-tls") pod "fe840242-fdf5-4733-8e08-c876f87401f2" (UID: "fe840242-fdf5-4733-8e08-c876f87401f2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:34:25.639512 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.639419 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fe840242-fdf5-4733-8e08-c876f87401f2" (UID: "fe840242-fdf5-4733-8e08-c876f87401f2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:34:25.639668 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.639651 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe840242-fdf5-4733-8e08-c876f87401f2" (UID: "fe840242-fdf5-4733-8e08-c876f87401f2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:34:25.639729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.639673 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "fe840242-fdf5-4733-8e08-c876f87401f2" (UID: "fe840242-fdf5-4733-8e08-c876f87401f2"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:34:25.738451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.738420 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fe840242-fdf5-4733-8e08-c876f87401f2-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:34:25.738451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.738448 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:34:25.738451 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.738456 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:34:25.738606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.738464 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:34:25.738606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.738472 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjgtg\" (UniqueName: \"kubernetes.io/projected/fe840242-fdf5-4733-8e08-c876f87401f2-kube-api-access-cjgtg\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:34:25.738606 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:25.738480 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fe840242-fdf5-4733-8e08-c876f87401f2-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:34:26.052404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.052371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" event={"ID":"fe840242-fdf5-4733-8e08-c876f87401f2","Type":"ContainerDied","Data":"0d6bc3b66a499c1b0adc1c730e5716a950e8053337dc8ebdf8365349a92fdedb"} Jul 28 12:34:26.052834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.052425 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj" Jul 28 12:34:26.052834 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.052429 2564 scope.go:117] "RemoveContainer" containerID="ed5d674ad6f67d48125830ed57e476e1cd83b762ec7f2c401815dc421cfd24b4" Jul 28 12:34:26.062222 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.062202 2564 scope.go:117] "RemoveContainer" containerID="bc2e15d41d02a63df9e3d12370e71bcaa96367aaf3ab6634fc7280a6bcd1c364" Jul 28 12:34:26.069728 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.069711 2564 scope.go:117] "RemoveContainer" containerID="c56022d473a450085fdb7654a6e0fdfb728584aca8a4194d82c9878e13a79731" Jul 28 12:34:26.076792 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.076770 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj"] Jul 28 12:34:26.077295 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.077270 2564 scope.go:117] "RemoveContainer" containerID="990354fbcf06585173964e33631bdd80ccc35ec8bd2660535494d2876c29f26f" Jul 28 12:34:26.080362 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.080343 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-s75rj"] Jul 28 12:34:26.084974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.084955 2564 scope.go:117] "RemoveContainer" containerID="677e7acb7a3f79bdac93731583993ab767973be084987a382c46aaabba355d6e" Jul 28 12:34:26.091972 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.091915 2564 scope.go:117] "RemoveContainer" containerID="4cee7078d83ddad0068d17bcb4b49bac54a23661dc4405402a2c14d7e69f9507" Jul 28 12:34:26.783444 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:34:26.783413 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" path="/var/lib/kubelet/pods/fe840242-fdf5-4733-8e08-c876f87401f2/volumes" Jul 28 12:35:16.593681 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.593647 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb"] Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594016 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594028 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594045 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="feast-apply" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594050 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="feast-apply" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594066 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="feast-init" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594075 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="feast-init" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594085 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="offline" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594091 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="offline" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594097 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594102 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594111 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="registry" Jul 28 12:35:16.594112 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594116 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="registry" Jul 28 12:35:16.594531 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594176 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="online" Jul 28 12:35:16.594531 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594186 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="offline" Jul 28 12:35:16.594531 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594194 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="ui" Jul 28 12:35:16.594531 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.594198 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe840242-fdf5-4733-8e08-c876f87401f2" containerName="registry" Jul 28 12:35:16.599697 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.599665 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.602809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.602785 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 28 12:35:16.602809 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.602804 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 28 12:35:16.603033 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.602828 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 28 12:35:16.603131 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.603113 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 28 12:35:16.603202 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.603148 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 28 12:35:16.603496 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.603475 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-tjq57\"" Jul 28 12:35:16.603597 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.603515 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 28 12:35:16.613397 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.613380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb"] Jul 28 12:35:16.724649 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.724625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6856eeb-59d7-4e66-96ea-88e040f1db7e-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.724751 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.724659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.724751 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.724706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.724751 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.724731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a6856eeb-59d7-4e66-96ea-88e040f1db7e-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.724751 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.724750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.724919 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.724806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwk8\" (UniqueName: \"kubernetes.io/projected/a6856eeb-59d7-4e66-96ea-88e040f1db7e-kube-api-access-6vwk8\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.825943 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.825919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6856eeb-59d7-4e66-96ea-88e040f1db7e-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.826037 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.825952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.826037 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.825998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.826037 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.826025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a6856eeb-59d7-4e66-96ea-88e040f1db7e-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.826169 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.826052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.826169 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.826082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwk8\" (UniqueName: \"kubernetes.io/projected/a6856eeb-59d7-4e66-96ea-88e040f1db7e-kube-api-access-6vwk8\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.826169 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:35:16.826151 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 12:35:16.826318 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:35:16.826198 2564 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 12:35:16.826318 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:35:16.826226 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls podName:a6856eeb-59d7-4e66-96ea-88e040f1db7e nodeName:}" failed. No retries permitted until 2026-07-28 12:35:17.32620803 +0000 UTC m=+1931.143137826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls") pod "feast-simple-feast-remote-setup-db6955c6c-r2nwb" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 12:35:16.826318 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:35:16.826264 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls podName:a6856eeb-59d7-4e66-96ea-88e040f1db7e nodeName:}" failed. No retries permitted until 2026-07-28 12:35:17.326246309 +0000 UTC m=+1931.143176106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls") pod "feast-simple-feast-remote-setup-db6955c6c-r2nwb" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 28 12:35:16.826446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.826316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6856eeb-59d7-4e66-96ea-88e040f1db7e-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.826644 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.826628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a6856eeb-59d7-4e66-96ea-88e040f1db7e-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.828442 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.828425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:16.834924 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:16.834905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwk8\" (UniqueName: \"kubernetes.io/projected/a6856eeb-59d7-4e66-96ea-88e040f1db7e-kube-api-access-6vwk8\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:17.330281 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:17.330247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:17.330281 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:17.330284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:17.332715 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:17.332685 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:17.332845 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:17.332811 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-r2nwb\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:17.509650 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:17.509618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:17.641334 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:17.641305 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb"] Jul 28 12:35:17.644290 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:35:17.644266 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6856eeb_59d7_4e66_96ea_88e040f1db7e.slice/crio-2413b45cba841d933af22b1b96bc2f055ea1b9d266287329a70455fceca37133 WatchSource:0}: Error finding container 2413b45cba841d933af22b1b96bc2f055ea1b9d266287329a70455fceca37133: Status 404 returned error can't find the container with id 2413b45cba841d933af22b1b96bc2f055ea1b9d266287329a70455fceca37133 Jul 28 12:35:18.268943 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:18.268907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerStarted","Data":"47f29cb954731440513700847cdb091f4a4f6f70d965121ae016b2bb477ded1c"} Jul 28 12:35:18.269119 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:18.268957 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerStarted","Data":"2413b45cba841d933af22b1b96bc2f055ea1b9d266287329a70455fceca37133"} Jul 28 12:35:23.291735 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:23.291705 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerID="47f29cb954731440513700847cdb091f4a4f6f70d965121ae016b2bb477ded1c" exitCode=0 Jul 28 12:35:23.292085 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:23.291771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerDied","Data":"47f29cb954731440513700847cdb091f4a4f6f70d965121ae016b2bb477ded1c"} Jul 28 12:35:24.297544 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:24.297504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerStarted","Data":"cceb2e5cea2ada98453e11d9ffcba24f6f16502403e3d118587b2c6ea13bf9ba"} Jul 28 12:35:29.321420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:29.321388 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerID="cceb2e5cea2ada98453e11d9ffcba24f6f16502403e3d118587b2c6ea13bf9ba" exitCode=0 Jul 28 12:35:29.321766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:29.321460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerDied","Data":"cceb2e5cea2ada98453e11d9ffcba24f6f16502403e3d118587b2c6ea13bf9ba"} Jul 28 12:35:30.329929 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:30.329864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerStarted","Data":"784b3e86e28692713f50e58dbaa1b4cfebead0df7ea340b6c04d5050fc743941"} Jul 28 12:35:30.330399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:30.329949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerStarted","Data":"2d193cd6f13fc87627fe22ae8e96988ca0012b3cca4770abedc32ebc4ba34f51"} Jul 28 12:35:30.330399 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:30.329969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerStarted","Data":"7acd96c68845c753b25e82a175efbdbffb8bf702d27bf7bf6e535e832f0b14c1"} Jul 28 12:35:30.355379 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:30.355318 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podStartSLOduration=14.355298032 podStartE2EDuration="14.355298032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:30.353466477 +0000 UTC m=+1944.170396295" watchObservedRunningTime="2026-07-28 12:35:30.355298032 +0000 UTC m=+1944.172227850" Jul 28 12:35:32.509984 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:32.509947 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:32.510418 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:32.510008 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:32.510418 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:32.510027 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:32.511524 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:32.511489 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Jul 28 12:35:32.511524 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:32.511505 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.36:8816: connect: connection refused" Jul 28 12:35:32.511732 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:32.511532 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 28 12:35:35.511115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:35.511085 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:35.511551 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:35.511179 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:35.511710 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:35.511685 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:35.511838 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:35.511731 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:35.512034 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:35.512020 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:35.512100 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:35.512074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:35.517625 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:35.517600 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:36.362943 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:36.362914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:35:36.367868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:35:36.367841 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:36:37.691420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.691390 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb"] Jul 28 12:36:37.691922 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.691608 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" containerID="cri-o://7acd96c68845c753b25e82a175efbdbffb8bf702d27bf7bf6e535e832f0b14c1" gracePeriod=30 Jul 28 12:36:37.691922 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.691663 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" containerID="cri-o://784b3e86e28692713f50e58dbaa1b4cfebead0df7ea340b6c04d5050fc743941" gracePeriod=30 Jul 28 12:36:37.691922 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.691687 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="offline" containerID="cri-o://2d193cd6f13fc87627fe22ae8e96988ca0012b3cca4770abedc32ebc4ba34f51" gracePeriod=30 Jul 28 12:36:37.842185 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.842155 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms"] Jul 28 12:36:37.842392 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.842363 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="registry" containerID="cri-o://ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58" gracePeriod=30 Jul 28 12:36:37.842528 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.842419 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" containerID="cri-o://1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec" gracePeriod=30 Jul 28 12:36:37.842528 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.842449 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" containerID="cri-o://10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd" gracePeriod=30 Jul 28 12:36:37.842528 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:37.842450 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="offline" containerID="cri-o://00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368" gracePeriod=30 Jul 28 12:36:38.628106 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.628079 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerID="784b3e86e28692713f50e58dbaa1b4cfebead0df7ea340b6c04d5050fc743941" exitCode=0 Jul 28 12:36:38.628106 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.628103 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerID="7acd96c68845c753b25e82a175efbdbffb8bf702d27bf7bf6e535e832f0b14c1" exitCode=0 Jul 28 12:36:38.628291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.628150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerDied","Data":"784b3e86e28692713f50e58dbaa1b4cfebead0df7ea340b6c04d5050fc743941"} Jul 28 12:36:38.628291 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.628187 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerDied","Data":"7acd96c68845c753b25e82a175efbdbffb8bf702d27bf7bf6e535e832f0b14c1"} Jul 28 12:36:38.631114 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.631092 2564 generic.go:356] "Generic (PLEG): container finished" podID="b098a990-1f03-4fe9-8389-769ee5f74283" containerID="1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec" exitCode=0 Jul 28 12:36:38.631114 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.631111 2564 generic.go:356] "Generic (PLEG): container finished" podID="b098a990-1f03-4fe9-8389-769ee5f74283" containerID="10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd" exitCode=0 Jul 28 12:36:38.631249 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.631131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerDied","Data":"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec"} Jul 28 12:36:38.631249 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:38.631156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerDied","Data":"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd"} Jul 28 12:36:43.605729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:43.605683 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Jul 28 12:36:43.998190 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:43.998154 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 12:36:45.512526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:45.512484 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Jul 28 12:36:46.363715 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:46.363670 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 28 12:36:53.605869 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:53.605819 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Jul 28 12:36:53.997947 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:53.997914 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 12:36:55.511768 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:55.511720 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Jul 28 12:36:56.363771 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:36:56.363728 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 28 12:37:03.606430 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:03.606377 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.35:8443: connect: connection refused" Jul 28 12:37:03.606940 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:03.606533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:37:03.998058 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:03.998021 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 12:37:03.998231 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:03.998147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:37:05.512432 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:05.512384 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.36:8443: connect: connection refused" Jul 28 12:37:05.512848 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:05.512544 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:37:06.363506 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:06.363458 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" probeResult="failure" output="Get \"https://10.134.0.36:6567/health\": dial tcp 10.134.0.36:6567: connect: connection refused" Jul 28 12:37:06.363721 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:06.363586 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:37:07.768315 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:07.768286 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerID="2d193cd6f13fc87627fe22ae8e96988ca0012b3cca4770abedc32ebc4ba34f51" exitCode=137 Jul 28 12:37:07.768604 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:07.768359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerDied","Data":"2d193cd6f13fc87627fe22ae8e96988ca0012b3cca4770abedc32ebc4ba34f51"} Jul 28 12:37:08.338135 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.338110 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:37:08.424385 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.424360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-offline-tls\") pod \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " Jul 28 12:37:08.424526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.424398 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vwk8\" (UniqueName: \"kubernetes.io/projected/a6856eeb-59d7-4e66-96ea-88e040f1db7e-kube-api-access-6vwk8\") pod \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " Jul 28 12:37:08.424526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.424434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a6856eeb-59d7-4e66-96ea-88e040f1db7e-registry-tls\") pod \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " Jul 28 12:37:08.424526 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.424493 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls\") pod \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " Jul 28 12:37:08.424687 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.424534 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls\") pod \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " Jul 28 12:37:08.424687 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.424579 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6856eeb-59d7-4e66-96ea-88e040f1db7e-feast-data\") pod \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\" (UID: \"a6856eeb-59d7-4e66-96ea-88e040f1db7e\") " Jul 28 12:37:08.424913 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.424850 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6856eeb-59d7-4e66-96ea-88e040f1db7e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a6856eeb-59d7-4e66-96ea-88e040f1db7e" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 12:37:08.425392 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.425368 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6856eeb-59d7-4e66-96ea-88e040f1db7e-feast-data" (OuterVolumeSpecName: "feast-data") pod "a6856eeb-59d7-4e66-96ea-88e040f1db7e" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:37:08.427255 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.427232 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a6856eeb-59d7-4e66-96ea-88e040f1db7e" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:08.427561 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.427527 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a6856eeb-59d7-4e66-96ea-88e040f1db7e" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:08.427647 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.427567 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls" (OuterVolumeSpecName: "online-tls") pod "a6856eeb-59d7-4e66-96ea-88e040f1db7e" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:08.427647 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.427592 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6856eeb-59d7-4e66-96ea-88e040f1db7e-kube-api-access-6vwk8" (OuterVolumeSpecName: "kube-api-access-6vwk8") pod "a6856eeb-59d7-4e66-96ea-88e040f1db7e" (UID: "a6856eeb-59d7-4e66-96ea-88e040f1db7e"). InnerVolumeSpecName "kube-api-access-6vwk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:37:08.477933 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.477915 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:37:08.525240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.525221 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6856eeb-59d7-4e66-96ea-88e040f1db7e-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.525240 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.525241 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.525404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.525250 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwk8\" (UniqueName: \"kubernetes.io/projected/a6856eeb-59d7-4e66-96ea-88e040f1db7e-kube-api-access-6vwk8\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.525404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.525258 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a6856eeb-59d7-4e66-96ea-88e040f1db7e-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.525404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.525267 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.525404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.525274 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6856eeb-59d7-4e66-96ea-88e040f1db7e-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.626493 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.626469 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-offline-tls\") pod \"b098a990-1f03-4fe9-8389-769ee5f74283\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " Jul 28 12:37:08.626669 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.626500 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlthh\" (UniqueName: \"kubernetes.io/projected/b098a990-1f03-4fe9-8389-769ee5f74283-kube-api-access-qlthh\") pod \"b098a990-1f03-4fe9-8389-769ee5f74283\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " Jul 28 12:37:08.626669 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.626548 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-online-tls\") pod \"b098a990-1f03-4fe9-8389-769ee5f74283\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " Jul 28 12:37:08.626669 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.626609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b098a990-1f03-4fe9-8389-769ee5f74283-feast-data\") pod \"b098a990-1f03-4fe9-8389-769ee5f74283\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " Jul 28 12:37:08.626669 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.626634 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-registry-tls\") pod \"b098a990-1f03-4fe9-8389-769ee5f74283\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " Jul 28 12:37:08.626669 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.626658 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-ui-tls\") pod \"b098a990-1f03-4fe9-8389-769ee5f74283\" (UID: \"b098a990-1f03-4fe9-8389-769ee5f74283\") " Jul 28 12:37:08.627210 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.627171 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b098a990-1f03-4fe9-8389-769ee5f74283-feast-data" (OuterVolumeSpecName: "feast-data") pod "b098a990-1f03-4fe9-8389-769ee5f74283" (UID: "b098a990-1f03-4fe9-8389-769ee5f74283"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:37:08.628632 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.628610 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b098a990-1f03-4fe9-8389-769ee5f74283" (UID: "b098a990-1f03-4fe9-8389-769ee5f74283"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:08.628738 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.628707 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b098a990-1f03-4fe9-8389-769ee5f74283-kube-api-access-qlthh" (OuterVolumeSpecName: "kube-api-access-qlthh") pod "b098a990-1f03-4fe9-8389-769ee5f74283" (UID: "b098a990-1f03-4fe9-8389-769ee5f74283"). InnerVolumeSpecName "kube-api-access-qlthh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:37:08.628791 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.628778 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b098a990-1f03-4fe9-8389-769ee5f74283" (UID: "b098a990-1f03-4fe9-8389-769ee5f74283"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:08.629217 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.629193 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-online-tls" (OuterVolumeSpecName: "online-tls") pod "b098a990-1f03-4fe9-8389-769ee5f74283" (UID: "b098a990-1f03-4fe9-8389-769ee5f74283"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:08.629313 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.629214 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b098a990-1f03-4fe9-8389-769ee5f74283" (UID: "b098a990-1f03-4fe9-8389-769ee5f74283"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 12:37:08.727729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.727676 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-online-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.727729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.727698 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b098a990-1f03-4fe9-8389-769ee5f74283-feast-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.727729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.727707 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.727729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.727714 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-ui-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.727729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.727721 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b098a990-1f03-4fe9-8389-769ee5f74283-offline-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.727729 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.727729 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlthh\" (UniqueName: \"kubernetes.io/projected/b098a990-1f03-4fe9-8389-769ee5f74283-kube-api-access-qlthh\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:08.775404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.775381 2564 generic.go:356] "Generic (PLEG): container finished" podID="b098a990-1f03-4fe9-8389-769ee5f74283" containerID="00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368" exitCode=137 Jul 28 12:37:08.775404 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.775403 2564 generic.go:356] "Generic (PLEG): container finished" podID="b098a990-1f03-4fe9-8389-769ee5f74283" containerID="ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58" exitCode=137 Jul 28 12:37:08.775775 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.775526 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" Jul 28 12:37:08.778292 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.778276 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" Jul 28 12:37:08.780177 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.780152 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerDied","Data":"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368"} Jul 28 12:37:08.780285 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.780198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerDied","Data":"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58"} Jul 28 12:37:08.780285 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.780221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms" event={"ID":"b098a990-1f03-4fe9-8389-769ee5f74283","Type":"ContainerDied","Data":"2c285d7378bf0a03f89b74559306d98407af64b393a5a3db60be8f1971fb462e"} Jul 28 12:37:08.780285 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.780243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb" event={"ID":"a6856eeb-59d7-4e66-96ea-88e040f1db7e","Type":"ContainerDied","Data":"2413b45cba841d933af22b1b96bc2f055ea1b9d266287329a70455fceca37133"} Jul 28 12:37:08.780285 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.780271 2564 scope.go:117] "RemoveContainer" containerID="1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec" Jul 28 12:37:08.790776 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.790758 2564 scope.go:117] "RemoveContainer" containerID="00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368" Jul 28 12:37:08.799022 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.799006 2564 scope.go:117] "RemoveContainer" containerID="10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd" Jul 28 12:37:08.803765 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.803744 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms"] Jul 28 12:37:08.807508 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.807489 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-h58ms"] Jul 28 12:37:08.807587 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.807490 2564 scope.go:117] "RemoveContainer" containerID="ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58" Jul 28 12:37:08.815195 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.815177 2564 scope.go:117] "RemoveContainer" containerID="635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20" Jul 28 12:37:08.819637 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.819618 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb"] Jul 28 12:37:08.823194 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.823157 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-r2nwb"] Jul 28 12:37:08.823255 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.823220 2564 scope.go:117] "RemoveContainer" containerID="a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953" Jul 28 12:37:08.830138 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.830122 2564 scope.go:117] "RemoveContainer" containerID="1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec" Jul 28 12:37:08.830378 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:08.830363 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec\": container with ID starting with 1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec not found: ID does not exist" containerID="1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec" Jul 28 12:37:08.830424 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.830386 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec"} err="failed to get container status \"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec\": rpc error: code = NotFound desc = could not find container \"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec\": container with ID starting with 1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec not found: ID does not exist" Jul 28 12:37:08.830424 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.830402 2564 scope.go:117] "RemoveContainer" containerID="00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368" Jul 28 12:37:08.830645 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:08.830628 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368\": container with ID starting with 00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368 not found: ID does not exist" containerID="00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368" Jul 28 12:37:08.830696 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.830650 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368"} err="failed to get container status \"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368\": rpc error: code = NotFound desc = could not find container \"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368\": container with ID starting with 00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368 not found: ID does not exist" Jul 28 12:37:08.830696 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.830667 2564 scope.go:117] "RemoveContainer" containerID="10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd" Jul 28 12:37:08.830930 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:08.830914 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd\": container with ID starting with 10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd not found: ID does not exist" containerID="10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd" Jul 28 12:37:08.830980 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.830935 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd"} err="failed to get container status \"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd\": rpc error: code = NotFound desc = could not find container \"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd\": container with ID starting with 10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd not found: ID does not exist" Jul 28 12:37:08.830980 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.830950 2564 scope.go:117] "RemoveContainer" containerID="ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58" Jul 28 12:37:08.831184 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:08.831171 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58\": container with ID starting with ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58 not found: ID does not exist" containerID="ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58" Jul 28 12:37:08.831225 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831186 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58"} err="failed to get container status \"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58\": rpc error: code = NotFound desc = could not find container \"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58\": container with ID starting with ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58 not found: ID does not exist" Jul 28 12:37:08.831225 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831202 2564 scope.go:117] "RemoveContainer" containerID="635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20" Jul 28 12:37:08.831402 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:08.831388 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20\": container with ID starting with 635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20 not found: ID does not exist" containerID="635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20" Jul 28 12:37:08.831446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831403 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20"} err="failed to get container status \"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20\": rpc error: code = NotFound desc = could not find container \"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20\": container with ID starting with 635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20 not found: ID does not exist" Jul 28 12:37:08.831446 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831416 2564 scope.go:117] "RemoveContainer" containerID="a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953" Jul 28 12:37:08.831581 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:08.831561 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953\": container with ID starting with a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953 not found: ID does not exist" containerID="a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953" Jul 28 12:37:08.831621 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831587 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953"} err="failed to get container status \"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953\": rpc error: code = NotFound desc = could not find container \"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953\": container with ID starting with a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953 not found: ID does not exist" Jul 28 12:37:08.831621 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831600 2564 scope.go:117] "RemoveContainer" containerID="1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec" Jul 28 12:37:08.831786 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831766 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec"} err="failed to get container status \"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec\": rpc error: code = NotFound desc = could not find container \"1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec\": container with ID starting with 1f41309bfffecdc8b6666cee2ffd1bea44fe7eb321cc69d3371e00aaac7595ec not found: ID does not exist" Jul 28 12:37:08.831855 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.831789 2564 scope.go:117] "RemoveContainer" containerID="00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368" Jul 28 12:37:08.832034 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832014 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368"} err="failed to get container status \"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368\": rpc error: code = NotFound desc = could not find container \"00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368\": container with ID starting with 00dc883388bb0d6a596f25bb68a78658a8e4e3e066a10ffb9571bb5e75968368 not found: ID does not exist" Jul 28 12:37:08.832082 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832034 2564 scope.go:117] "RemoveContainer" containerID="10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd" Jul 28 12:37:08.832251 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832234 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd"} err="failed to get container status \"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd\": rpc error: code = NotFound desc = could not find container \"10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd\": container with ID starting with 10319e6bb3211a9cd9bca2417821b9cdb80c578101560f63f2340473666d7bfd not found: ID does not exist" Jul 28 12:37:08.832303 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832251 2564 scope.go:117] "RemoveContainer" containerID="ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58" Jul 28 12:37:08.832422 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832405 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58"} err="failed to get container status \"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58\": rpc error: code = NotFound desc = could not find container \"ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58\": container with ID starting with ca23a680aa73db0b5fa8c3f74002b85f29fa3691abb0721cb2e9c1d40828be58 not found: ID does not exist" Jul 28 12:37:08.832469 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832426 2564 scope.go:117] "RemoveContainer" containerID="635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20" Jul 28 12:37:08.832620 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832605 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20"} err="failed to get container status \"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20\": rpc error: code = NotFound desc = could not find container \"635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20\": container with ID starting with 635bdbc9c74d9500316102d0957ed9e3746b947dceff9cb5f58aaf0764bd6d20 not found: ID does not exist" Jul 28 12:37:08.832620 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832620 2564 scope.go:117] "RemoveContainer" containerID="a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953" Jul 28 12:37:08.832850 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832832 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953"} err="failed to get container status \"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953\": rpc error: code = NotFound desc = could not find container \"a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953\": container with ID starting with a7973945f319368fedc575ef1bc26d1b2fc7feacc6177d2f98e8f0b3ff4e4953 not found: ID does not exist" Jul 28 12:37:08.832941 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.832852 2564 scope.go:117] "RemoveContainer" containerID="784b3e86e28692713f50e58dbaa1b4cfebead0df7ea340b6c04d5050fc743941" Jul 28 12:37:08.839868 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.839854 2564 scope.go:117] "RemoveContainer" containerID="2d193cd6f13fc87627fe22ae8e96988ca0012b3cca4770abedc32ebc4ba34f51" Jul 28 12:37:08.846939 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.846906 2564 scope.go:117] "RemoveContainer" containerID="7acd96c68845c753b25e82a175efbdbffb8bf702d27bf7bf6e535e832f0b14c1" Jul 28 12:37:08.853558 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.853543 2564 scope.go:117] "RemoveContainer" containerID="cceb2e5cea2ada98453e11d9ffcba24f6f16502403e3d118587b2c6ea13bf9ba" Jul 28 12:37:08.860388 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:08.860375 2564 scope.go:117] "RemoveContainer" containerID="47f29cb954731440513700847cdb091f4a4f6f70d965121ae016b2bb477ded1c" Jul 28 12:37:10.780940 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:10.780870 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" path="/var/lib/kubelet/pods/a6856eeb-59d7-4e66-96ea-88e040f1db7e/volumes" Jul 28 12:37:10.781491 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:10.781472 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" path="/var/lib/kubelet/pods/b098a990-1f03-4fe9-8389-769ee5f74283/volumes" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.264816 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zs4x7/must-gather-kxfn6"] Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265549 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="offline" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265568 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="offline" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265599 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="feast-init" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265608 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="feast-init" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265631 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="feast-init" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265639 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="feast-init" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265649 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="registry" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265657 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="registry" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265672 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265681 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265700 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="offline" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265709 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="offline" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265738 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265747 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265774 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="feast-apply" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265782 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="feast-apply" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265792 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265800 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265820 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="feast-apply" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265829 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="feast-apply" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265849 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.265857 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.266007 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="online" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.266025 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="online" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.266034 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="offline" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.266046 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="ui" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.266061 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="offline" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.266076 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b098a990-1f03-4fe9-8389-769ee5f74283" containerName="registry" Jul 28 12:37:27.267974 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.266084 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6856eeb-59d7-4e66-96ea-88e040f1db7e" containerName="ui" Jul 28 12:37:27.272309 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.272290 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.274679 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.274657 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zs4x7\"/\"kube-root-ca.crt\"" Jul 28 12:37:27.275585 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.275568 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zs4x7\"/\"openshift-service-ca.crt\"" Jul 28 12:37:27.275680 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.275583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zs4x7\"/\"default-dockercfg-lb6pl\"" Jul 28 12:37:27.278565 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.278546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zs4x7/must-gather-kxfn6"] Jul 28 12:37:27.369177 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.369152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-must-gather-output\") pod \"must-gather-kxfn6\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.369296 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.369193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrpn\" (UniqueName: \"kubernetes.io/projected/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-kube-api-access-lqrpn\") pod \"must-gather-kxfn6\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.470501 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.470479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-must-gather-output\") pod \"must-gather-kxfn6\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.470600 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.470518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrpn\" (UniqueName: \"kubernetes.io/projected/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-kube-api-access-lqrpn\") pod \"must-gather-kxfn6\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.470773 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.470757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-must-gather-output\") pod \"must-gather-kxfn6\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.479534 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.479513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrpn\" (UniqueName: \"kubernetes.io/projected/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-kube-api-access-lqrpn\") pod \"must-gather-kxfn6\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.596279 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.596228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:27.721120 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.721092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zs4x7/must-gather-kxfn6"] Jul 28 12:37:27.724761 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:37:27.724734 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb013c55_4ef4_4bb4_9f5a_3c1729c1f381.slice/crio-d98d85933df94ab8cb7f9a25512255e462e2c661a4a82d3c8a6ea42ff6d20887 WatchSource:0}: Error finding container d98d85933df94ab8cb7f9a25512255e462e2c661a4a82d3c8a6ea42ff6d20887: Status 404 returned error can't find the container with id d98d85933df94ab8cb7f9a25512255e462e2c661a4a82d3c8a6ea42ff6d20887 Jul 28 12:37:27.726969 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.726955 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 12:37:27.857819 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:27.857759 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" event={"ID":"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381","Type":"ContainerStarted","Data":"d98d85933df94ab8cb7f9a25512255e462e2c661a4a82d3c8a6ea42ff6d20887"} Jul 28 12:37:34.893283 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:34.893242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" event={"ID":"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381","Type":"ContainerStarted","Data":"414f7666aef4a95a114eb2fffa3cb619f4e8d46b0510cc6be63b5f84fe49f5a0"} Jul 28 12:37:34.893664 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:34.893302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" event={"ID":"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381","Type":"ContainerStarted","Data":"df3841eb3e107a4d1d25c7efbe1a103898a3a5a89443347e4e3ad81e7eba6d11"} Jul 28 12:37:34.909930 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:34.909866 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" podStartSLOduration=1.698340618 podStartE2EDuration="7.90985111s" totalImagesPullingTime="6.211510492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:37:34.908397338 +0000 UTC m=+2068.725327159" watchObservedRunningTime="2026-07-28 12:37:34.90985111 +0000 UTC m=+2068.726780927" Jul 28 12:37:43.929830 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:43.929795 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerID="df3841eb3e107a4d1d25c7efbe1a103898a3a5a89443347e4e3ad81e7eba6d11" exitCode=0 Jul 28 12:37:43.930228 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:43.929867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" event={"ID":"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381","Type":"ContainerDied","Data":"df3841eb3e107a4d1d25c7efbe1a103898a3a5a89443347e4e3ad81e7eba6d11"} Jul 28 12:37:43.930228 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:43.930115 2564 scope.go:117] "RemoveContainer" containerID="df3841eb3e107a4d1d25c7efbe1a103898a3a5a89443347e4e3ad81e7eba6d11" Jul 28 12:37:44.416815 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:44.416784 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zs4x7_must-gather-kxfn6_cb013c55-4ef4-4bb4-9f5a-3c1729c1f381/gather/0.log" Jul 28 12:37:47.849271 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:47.849245 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4mk2d_caf368fa-46b2-4bbc-a2a0-d3f35022719a/global-pull-secret-syncer/0.log" Jul 28 12:37:48.077774 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:48.077747 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wr2z4_5d522fb7-96d0-40b8-aa15-33ca968f946c/konnectivity-agent/0.log" Jul 28 12:37:48.169760 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:48.169693 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-8.ec2.internal_5b4c93f4e70558b77ddac96ac849afbc/haproxy/0.log" Jul 28 12:37:49.778858 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:49.778829 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zs4x7/must-gather-kxfn6"] Jul 28 12:37:49.779263 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:49.779019 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerName="copy" containerID="cri-o://414f7666aef4a95a114eb2fffa3cb619f4e8d46b0510cc6be63b5f84fe49f5a0" gracePeriod=2 Jul 28 12:37:49.785911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:49.785864 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zs4x7/must-gather-kxfn6"] Jul 28 12:37:49.786544 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:49.786511 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-zs4x7\"/\"must-gather-kxfn6\": pods \"must-gather-kxfn6\" not found" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:49.958437 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:49.958363 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zs4x7_must-gather-kxfn6_cb013c55-4ef4-4bb4-9f5a-3c1729c1f381/copy/0.log" Jul 28 12:37:49.958766 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:49.958744 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerID="414f7666aef4a95a114eb2fffa3cb619f4e8d46b0510cc6be63b5f84fe49f5a0" exitCode=143 Jul 28 12:37:50.005906 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.005871 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zs4x7_must-gather-kxfn6_cb013c55-4ef4-4bb4-9f5a-3c1729c1f381/copy/0.log" Jul 28 12:37:50.007403 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.007388 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:50.009431 ip-10-0-134-8 kubenswrapper[2564]: E0728 12:37:50.009410 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kxfn6\" is forbidden: User \"system:node:ip-10-0-134-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-zs4x7\": no relationship found between node 'ip-10-0-134-8.ec2.internal' and this object" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:50.052121 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.052072 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqrpn\" (UniqueName: \"kubernetes.io/projected/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-kube-api-access-lqrpn\") pod \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " Jul 28 12:37:50.052208 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.052147 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-must-gather-output\") pod \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\" (UID: \"cb013c55-4ef4-4bb4-9f5a-3c1729c1f381\") " Jul 28 12:37:50.052463 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.052444 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" (UID: "cb013c55-4ef4-4bb4-9f5a-3c1729c1f381"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 12:37:50.054302 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.054283 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-kube-api-access-lqrpn" (OuterVolumeSpecName: "kube-api-access-lqrpn") pod "cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" (UID: "cb013c55-4ef4-4bb4-9f5a-3c1729c1f381"). InnerVolumeSpecName "kube-api-access-lqrpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 12:37:50.152993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.152963 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrpn\" (UniqueName: \"kubernetes.io/projected/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-kube-api-access-lqrpn\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:50.152993 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.152987 2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381-must-gather-output\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\"" Jul 28 12:37:50.779518 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.779488 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" path="/var/lib/kubelet/pods/cb013c55-4ef4-4bb4-9f5a-3c1729c1f381/volumes" Jul 28 12:37:50.963742 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.963717 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zs4x7_must-gather-kxfn6_cb013c55-4ef4-4bb4-9f5a-3c1729c1f381/copy/0.log" Jul 28 12:37:50.964146 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.964130 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zs4x7/must-gather-kxfn6" Jul 28 12:37:50.964223 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.964157 2564 scope.go:117] "RemoveContainer" containerID="414f7666aef4a95a114eb2fffa3cb619f4e8d46b0510cc6be63b5f84fe49f5a0" Jul 28 12:37:50.972115 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:50.972097 2564 scope.go:117] "RemoveContainer" containerID="df3841eb3e107a4d1d25c7efbe1a103898a3a5a89443347e4e3ad81e7eba6d11" Jul 28 12:37:52.042483 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.042455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_efd872fc-5cef-4712-b468-3ea71bcd14f2/alertmanager/0.log" Jul 28 12:37:52.069398 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.069372 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_efd872fc-5cef-4712-b468-3ea71bcd14f2/config-reloader/0.log" Jul 28 12:37:52.099934 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.099912 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_efd872fc-5cef-4712-b468-3ea71bcd14f2/kube-rbac-proxy-web/0.log" Jul 28 12:37:52.127357 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.127322 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_efd872fc-5cef-4712-b468-3ea71bcd14f2/kube-rbac-proxy/0.log" Jul 28 12:37:52.154713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.154690 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_efd872fc-5cef-4712-b468-3ea71bcd14f2/kube-rbac-proxy-metric/0.log" Jul 28 12:37:52.190448 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.190430 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_efd872fc-5cef-4712-b468-3ea71bcd14f2/prom-label-proxy/0.log" Jul 28 12:37:52.216443 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.216420 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_efd872fc-5cef-4712-b468-3ea71bcd14f2/init-config-reloader/0.log" Jul 28 12:37:52.402271 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.402208 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-278xn_5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9/kube-state-metrics/0.log" Jul 28 12:37:52.422678 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.422659 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-278xn_5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9/kube-rbac-proxy-main/0.log" Jul 28 12:37:52.446098 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.446073 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-278xn_5cea282d-37c7-4a3a-a1ae-23e65b1bdeb9/kube-rbac-proxy-self/0.log" Jul 28 12:37:52.481090 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.481067 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-768c457df4-csh9b_ec989cab-4f7d-459a-8002-38978bd5636a/metrics-server/0.log" Jul 28 12:37:52.516534 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.516511 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-mv4k4_a90f56d7-67cd-4a18-8464-95f197f4d8c3/monitoring-plugin/0.log" Jul 28 12:37:52.639448 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.639429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtw7p_d1a0f45c-2485-44cc-beae-35eb28bd86d1/node-exporter/0.log" Jul 28 12:37:52.664445 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.664387 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtw7p_d1a0f45c-2485-44cc-beae-35eb28bd86d1/kube-rbac-proxy/0.log" Jul 28 12:37:52.687635 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.687617 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtw7p_d1a0f45c-2485-44cc-beae-35eb28bd86d1/init-textfile/0.log" Jul 28 12:37:52.799566 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.799529 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-vfd2b_c351e7c4-094e-4393-a391-0fcb3e67ed1d/kube-rbac-proxy-main/0.log" Jul 28 12:37:52.825517 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.825498 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-vfd2b_c351e7c4-094e-4393-a391-0fcb3e67ed1d/kube-rbac-proxy-self/0.log" Jul 28 12:37:52.851490 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.851472 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-vfd2b_c351e7c4-094e-4393-a391-0fcb3e67ed1d/openshift-state-metrics/0.log" Jul 28 12:37:52.901006 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.900981 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95ac02ce-43e4-4f24-9bad-f5936f09028c/prometheus/0.log" Jul 28 12:37:52.931161 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.931114 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95ac02ce-43e4-4f24-9bad-f5936f09028c/config-reloader/0.log" Jul 28 12:37:52.955262 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.955245 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95ac02ce-43e4-4f24-9bad-f5936f09028c/thanos-sidecar/0.log" Jul 28 12:37:52.983465 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:52.983446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95ac02ce-43e4-4f24-9bad-f5936f09028c/kube-rbac-proxy-web/0.log" Jul 28 12:37:53.008142 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.008123 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95ac02ce-43e4-4f24-9bad-f5936f09028c/kube-rbac-proxy/0.log" Jul 28 12:37:53.031138 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.031123 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95ac02ce-43e4-4f24-9bad-f5936f09028c/kube-rbac-proxy-thanos/0.log" Jul 28 12:37:53.065122 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.065100 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95ac02ce-43e4-4f24-9bad-f5936f09028c/init-config-reloader/0.log" Jul 28 12:37:53.100425 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.100390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-f2vhq_cd4840bc-966d-4609-b62f-2246082309ed/prometheus-operator/0.log" Jul 28 12:37:53.139685 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.139659 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-f2vhq_cd4840bc-966d-4609-b62f-2246082309ed/kube-rbac-proxy/0.log" Jul 28 12:37:53.177271 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.177254 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-8hn4b_606a9748-163b-4e92-bbcd-9c3ab4d3ce91/prometheus-operator-admission-webhook/0.log" Jul 28 12:37:53.217675 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.217658 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b4996fbf9-dmx8d_e64e69a2-05f2-43a3-a02b-226b17f00515/telemeter-client/0.log" Jul 28 12:37:53.253654 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.253636 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b4996fbf9-dmx8d_e64e69a2-05f2-43a3-a02b-226b17f00515/reload/0.log" Jul 28 12:37:53.286317 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.286300 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b4996fbf9-dmx8d_e64e69a2-05f2-43a3-a02b-226b17f00515/kube-rbac-proxy/0.log" Jul 28 12:37:53.324397 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.324380 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c556dff8f-52c5j_179ffc7c-5199-4cc9-9a62-64678a1f4449/thanos-query/0.log" Jul 28 12:37:53.349911 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.349872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c556dff8f-52c5j_179ffc7c-5199-4cc9-9a62-64678a1f4449/kube-rbac-proxy-web/0.log" Jul 28 12:37:53.379690 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.379669 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c556dff8f-52c5j_179ffc7c-5199-4cc9-9a62-64678a1f4449/kube-rbac-proxy/0.log" Jul 28 12:37:53.404439 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.404423 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c556dff8f-52c5j_179ffc7c-5199-4cc9-9a62-64678a1f4449/prom-label-proxy/0.log" Jul 28 12:37:53.437462 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.437445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c556dff8f-52c5j_179ffc7c-5199-4cc9-9a62-64678a1f4449/kube-rbac-proxy-rules/0.log" Jul 28 12:37:53.466008 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:53.465986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c556dff8f-52c5j_179ffc7c-5199-4cc9-9a62-64678a1f4449/kube-rbac-proxy-metrics/0.log" Jul 28 12:37:55.514100 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.514070 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn"] Jul 28 12:37:55.514461 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.514407 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerName="gather" Jul 28 12:37:55.514461 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.514417 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerName="gather" Jul 28 12:37:55.514461 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.514427 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerName="copy" Jul 28 12:37:55.514461 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.514432 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerName="copy" Jul 28 12:37:55.514584 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.514489 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerName="copy" Jul 28 12:37:55.514584 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.514498 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb013c55-4ef4-4bb4-9f5a-3c1729c1f381" containerName="gather" Jul 28 12:37:55.518094 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.518079 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.520465 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.520444 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4tfnx\"/\"kube-root-ca.crt\"" Jul 28 12:37:55.520802 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.520787 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4tfnx\"/\"openshift-service-ca.crt\"" Jul 28 12:37:55.521420 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.521404 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4tfnx\"/\"default-dockercfg-trrcn\"" Jul 28 12:37:55.526259 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.526240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn"] Jul 28 12:37:55.596062 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.596039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-proc\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.596162 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.596080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-podres\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.596162 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.596117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-lib-modules\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.596247 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.596201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6x5\" (UniqueName: \"kubernetes.io/projected/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-kube-api-access-4b6x5\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.596247 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.596228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-sys\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696651 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6x5\" (UniqueName: \"kubernetes.io/projected/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-kube-api-access-4b6x5\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696736 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-sys\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696736 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-proc\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696736 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-podres\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696898 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-lib-modules\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696898 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-sys\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696898 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-proc\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.696898 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696870 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-podres\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.697036 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.696914 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-lib-modules\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.705005 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.704985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6x5\" (UniqueName: \"kubernetes.io/projected/5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12-kube-api-access-4b6x5\") pod \"perf-node-gather-daemonset-jc4jn\" (UID: \"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12\") " pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.828134 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.828073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:55.955705 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.955684 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn"] Jul 28 12:37:55.958201 ip-10-0-134-8 kubenswrapper[2564]: W0728 12:37:55.958170 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ba32f2b_4e7d_4de2_9bb5_80e49b7c1d12.slice/crio-ae98bc0c205850ea0de31e47710808b0ed6b65254b7f8d0db4577f3238ed6b31 WatchSource:0}: Error finding container ae98bc0c205850ea0de31e47710808b0ed6b65254b7f8d0db4577f3238ed6b31: Status 404 returned error can't find the container with id ae98bc0c205850ea0de31e47710808b0ed6b65254b7f8d0db4577f3238ed6b31 Jul 28 12:37:55.983623 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:55.983597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" event={"ID":"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12","Type":"ContainerStarted","Data":"ae98bc0c205850ea0de31e47710808b0ed6b65254b7f8d0db4577f3238ed6b31"} Jul 28 12:37:56.894211 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:56.894170 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kr5xt_d3297c9b-23e3-49fc-8b8e-18d121542221/dns/0.log" Jul 28 12:37:56.937018 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:56.936989 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kr5xt_d3297c9b-23e3-49fc-8b8e-18d121542221/kube-rbac-proxy/0.log" Jul 28 12:37:56.988504 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:56.988477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" event={"ID":"5ba32f2b-4e7d-4de2-9bb5-80e49b7c1d12","Type":"ContainerStarted","Data":"8df64cec9b4cc78e41925ff0004818a47d631082e295dbea4bbbbe288a7409be"} Jul 28 12:37:56.988621 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:56.988582 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:37:57.004923 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:57.004858 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" podStartSLOduration=2.004844823 podStartE2EDuration="2.004844823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:37:57.004195652 +0000 UTC m=+2090.821125472" watchObservedRunningTime="2026-07-28 12:37:57.004844823 +0000 UTC m=+2090.821774697" Jul 28 12:37:57.154652 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:57.154581 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w85q8_1ea75a61-bc22-494b-8843-902854593d6c/dns-node-resolver/0.log" Jul 28 12:37:57.661138 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:57.661108 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sh2lt_4e57a037-dd6f-4793-bfa6-6145355fa80f/node-ca/0.log" Jul 28 12:37:58.810852 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:58.810820 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vqt77_bae8697e-e6b5-43c7-93e5-e5d8c5ae2c42/serve-healthcheck-canary/0.log" Jul 28 12:37:59.322703 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:59.322662 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-trjwp_aec5250b-4ff2-414b-aab2-a8c726a09765/kube-rbac-proxy/0.log" Jul 28 12:37:59.344118 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:59.344089 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-trjwp_aec5250b-4ff2-414b-aab2-a8c726a09765/exporter/0.log" Jul 28 12:37:59.366141 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:37:59.366120 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-trjwp_aec5250b-4ff2-414b-aab2-a8c726a09765/extractor/0.log" Jul 28 12:38:03.002447 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:03.002416 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4tfnx/perf-node-gather-daemonset-jc4jn" Jul 28 12:38:04.988942 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:04.988914 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnjtb_6abe718b-42b3-4ae8-8051-8bb46abd4db4/kube-multus-additional-cni-plugins/0.log" Jul 28 12:38:05.010758 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.010728 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnjtb_6abe718b-42b3-4ae8-8051-8bb46abd4db4/egress-router-binary-copy/0.log" Jul 28 12:38:05.036640 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.036620 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnjtb_6abe718b-42b3-4ae8-8051-8bb46abd4db4/cni-plugins/0.log" Jul 28 12:38:05.062949 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.062921 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnjtb_6abe718b-42b3-4ae8-8051-8bb46abd4db4/bond-cni-plugin/0.log" Jul 28 12:38:05.085510 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.085492 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnjtb_6abe718b-42b3-4ae8-8051-8bb46abd4db4/routeoverride-cni/0.log" Jul 28 12:38:05.106841 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.106828 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnjtb_6abe718b-42b3-4ae8-8051-8bb46abd4db4/whereabouts-cni-bincopy/0.log" Jul 28 12:38:05.127534 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.127519 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wnjtb_6abe718b-42b3-4ae8-8051-8bb46abd4db4/whereabouts-cni/0.log" Jul 28 12:38:05.371556 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.371532 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wb5fb_ff2fb42a-86c5-4534-a6b7-88d8f2a5d1a5/kube-multus/0.log" Jul 28 12:38:05.454915 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.454897 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gklcw_0afc5b7d-6f03-4e22-b639-88fa243b94d3/network-metrics-daemon/0.log" Jul 28 12:38:05.474847 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:05.474830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gklcw_0afc5b7d-6f03-4e22-b639-88fa243b94d3/kube-rbac-proxy/0.log" Jul 28 12:38:06.778197 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:06.778081 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:38:06.794582 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:06.781590 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:38:06.915984 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:06.915961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-controller/0.log" Jul 28 12:38:06.933608 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:06.933586 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/1.log" Jul 28 12:38:06.933734 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:06.933622 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovn-acl-logging/0.log" Jul 28 12:38:06.954343 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:06.954317 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/kube-rbac-proxy-node/0.log" Jul 28 12:38:06.977713 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:06.977679 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 12:38:07.001280 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:07.001253 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/northd/0.log" Jul 28 12:38:07.023888 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:07.023852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/nbdb/0.log" Jul 28 12:38:07.051407 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:07.051391 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/sbdb/0.log" Jul 28 12:38:07.122183 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:07.122157 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7967_71f8cf88-2dcc-4d71-a25b-b9bd281e00e1/ovnkube-controller/0.log" Jul 28 12:38:08.215090 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:08.215065 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hmlhv_12686a8e-d528-4138-a8ff-0eaf60244bd2/network-check-target-container/0.log" Jul 28 12:38:09.178780 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:09.178750 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-ndsld_dcf5bafc-60f8-4a8e-964c-b697c819f0f4/iptables-alerter/0.log" Jul 28 12:38:09.815633 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:09.815611 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-82vhr_37aa38d5-5f42-4c1a-b3ab-82fe91a37aca/tuned/0.log" Jul 28 12:38:13.413180 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:13.413148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-nh7zz_6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b/csi-driver/0.log" Jul 28 12:38:13.437305 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:13.437268 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-nh7zz_6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b/csi-node-driver-registrar/0.log" Jul 28 12:38:13.461126 ip-10-0-134-8 kubenswrapper[2564]: I0728 12:38:13.461105 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-nh7zz_6fe798e7-ffbf-48c7-9ebf-3bc09e47ba8b/csi-liveness-probe/0.log"