Jul 22 16:09:58.606648 ip-10-0-142-1 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 16:09:58.606657 ip-10-0-142-1 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 16:09:58.606664 ip-10-0-142-1 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 16:09:58.606901 ip-10-0-142-1 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 16:10:08.671676 ip-10-0-142-1 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 16:10:08.671698 ip-10-0-142-1 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 00ba25d8790b49bfa6a482c62f16754a -- Jul 22 16:12:03.725171 ip-10-0-142-1 systemd[1]: Starting Kubernetes Kubelet... Jul 22 16:12:04.061187 ip-10-0-142-1 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:12:04.061187 ip-10-0-142-1 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 16:12:04.061187 ip-10-0-142-1 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:12:04.061187 ip-10-0-142-1 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 16:12:04.061187 ip-10-0-142-1 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:12:04.062010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.061936 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 16:12:04.065828 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065813 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:12:04.065828 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065828 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065832 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065836 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065839 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065842 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065845 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065848 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065851 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065853 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065856 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065859 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065862 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065864 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065867 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065870 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065872 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065890 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065894 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065898 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065910 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:12:04.065954 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065913 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065917 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065920 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065923 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065926 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065929 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065932 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065935 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065937 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065940 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065943 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065945 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065948 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065951 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065954 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065957 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065960 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065963 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065966 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065968 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:12:04.066442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065973 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065977 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065980 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065982 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065985 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065987 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065990 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065993 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065996 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.065999 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066001 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066004 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066007 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066010 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066012 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066016 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066018 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066021 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066024 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066026 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:12:04.066920 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066029 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066033 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066037 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066041 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066044 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066047 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066050 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066053 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066056 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066058 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066063 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066065 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066068 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066071 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066073 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066076 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066079 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066081 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066084 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:12:04.067388 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066087 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066090 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066093 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066095 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066098 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066101 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066104 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066106 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066109 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066111 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066114 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066116 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066119 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066122 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066124 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066127 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066130 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066133 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066135 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066796 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:12:04.067839 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066802 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066805 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066808 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066811 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066813 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066816 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066818 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066821 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066824 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066826 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066829 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066831 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066834 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066838 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066841 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066843 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066846 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066849 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066851 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066854 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:12:04.068322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066856 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066859 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066862 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066864 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066868 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066872 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066875 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066892 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066895 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066898 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066902 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066906 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066909 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066912 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066915 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066918 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066921 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066923 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066926 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066929 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:12:04.068830 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066932 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066934 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066937 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066939 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066942 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066944 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066947 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066950 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066954 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066956 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066959 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066961 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066964 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066967 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066969 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066972 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066974 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066976 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066979 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066982 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:12:04.069323 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066984 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066987 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066990 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066992 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066995 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066997 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.066999 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067002 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067004 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067007 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067009 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067012 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067014 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067017 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067021 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067024 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067027 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067030 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067033 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:12:04.069795 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067038 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067041 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067044 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067047 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067050 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067052 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067055 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067059 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067061 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067064 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067067 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067070 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067073 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067075 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067078 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067080 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067083 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067085 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067088 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:12:04.070261 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067420 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067424 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067427 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067430 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067432 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067435 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067437 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067440 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067442 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067445 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067447 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067450 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067452 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067455 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067458 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067461 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067463 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067466 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067469 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:12:04.070730 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067471 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067474 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067476 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067479 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067482 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067484 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067487 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067490 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067494 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067497 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067500 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067503 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067506 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067508 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067511 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067513 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067516 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067518 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067520 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067523 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:12:04.071192 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067525 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067528 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067530 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067533 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067535 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067538 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067540 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067544 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067546 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067549 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067551 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067554 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067557 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067559 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067562 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067566 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067570 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067573 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067576 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:12:04.071649 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067578 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067582 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067586 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067589 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067592 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067594 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067597 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067599 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067602 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067604 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067607 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067610 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067612 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067615 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067617 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067619 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067622 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067624 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067627 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:12:04.072251 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067630 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067635 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067638 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067641 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067643 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067646 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067648 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067651 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067654 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067657 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067660 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067662 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067665 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067667 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067670 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067672 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067675 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067678 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067680 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067683 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:12:04.073041 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067686 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.067688 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067765 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067774 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067784 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067791 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067798 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067802 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067806 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067810 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067813 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067816 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067819 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067824 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067827 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067831 2578 flags.go:64] FLAG: --cgroup-root="" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067834 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067837 2578 flags.go:64] FLAG: --client-ca-file="" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067839 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067843 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067848 2578 flags.go:64] FLAG: --cluster-domain="" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067851 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067854 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067857 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 16:12:04.073857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067860 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067864 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067867 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067870 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067873 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067888 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067892 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067894 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067898 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067901 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067905 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067908 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067910 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067913 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067916 2578 flags.go:64] FLAG: --enable-server="true" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067919 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067923 2578 flags.go:64] FLAG: --event-burst="100" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067926 2578 flags.go:64] FLAG: --event-qps="50" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067929 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067932 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067935 2578 flags.go:64] FLAG: --eviction-hard="" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067940 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067943 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067946 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067949 2578 flags.go:64] FLAG: --eviction-soft="" Jul 22 16:12:04.074689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067952 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067955 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067959 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067963 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067966 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067969 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067971 2578 flags.go:64] FLAG: --feature-gates="" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067975 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067978 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067981 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067984 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067987 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067990 2578 flags.go:64] FLAG: --help="false" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067993 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-1.ec2.internal" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067996 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.067999 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068002 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068005 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068009 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068012 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068015 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068018 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068020 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 16:12:04.075293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068023 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068026 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068029 2578 flags.go:64] FLAG: --kube-reserved="" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068032 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068035 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068039 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068042 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068045 2578 flags.go:64] FLAG: --lock-file="" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068048 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068051 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068054 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068059 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068063 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068066 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068069 2578 flags.go:64] FLAG: --logging-format="text" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068072 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068075 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068078 2578 flags.go:64] FLAG: --manifest-url="" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068081 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068085 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068088 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068091 2578 flags.go:64] FLAG: --max-pods="110" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068094 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068097 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068100 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 16:12:04.075855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068103 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068106 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068109 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068112 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068118 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068121 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068124 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068127 2578 flags.go:64] FLAG: --pod-cidr="" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068130 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068135 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068138 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068141 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068145 2578 flags.go:64] FLAG: --port="10250" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068148 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068151 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da37725641688a87" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068154 2578 flags.go:64] FLAG: --qos-reserved="" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068157 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068160 2578 flags.go:64] FLAG: --register-node="true" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068163 2578 flags.go:64] FLAG: --register-with-taints="" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068167 2578 flags.go:64] FLAG: --registry-burst="10" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068171 2578 flags.go:64] FLAG: --registry-qps="5" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068174 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068177 2578 flags.go:64] FLAG: --reserved-memory="" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068180 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068183 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 16:12:04.076468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068186 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068189 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068192 2578 flags.go:64] FLAG: --runonce="false" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068195 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068197 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068201 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068203 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068206 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068210 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068213 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068216 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068218 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068221 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068224 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068227 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068230 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068233 2578 flags.go:64] FLAG: --system-cgroups="" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068236 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068241 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068245 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068248 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068252 2578 flags.go:64] FLAG: --tls-min-version="" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068254 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068257 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068260 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 16:12:04.077109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068263 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068266 2578 flags.go:64] FLAG: --v="2" Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068271 2578 flags.go:64] FLAG: --version="false" Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068275 2578 flags.go:64] FLAG: --vmodule="" Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068278 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.068282 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068387 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068390 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068393 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068396 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068398 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068402 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068404 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068407 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068409 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068412 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068415 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068418 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068420 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068423 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068426 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:12:04.077964 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068429 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068431 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068434 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068436 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068439 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068442 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068445 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068447 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068450 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068453 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068455 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068458 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068461 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068464 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068467 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068469 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068472 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068474 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068477 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:12:04.078816 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068479 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068482 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068485 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068488 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068490 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068493 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068496 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068498 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068501 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068503 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068506 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068508 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068511 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068513 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068516 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068518 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068521 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068523 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068527 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068530 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:12:04.079385 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068532 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068535 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068537 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068539 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068542 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068544 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068548 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068552 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068556 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068559 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068562 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068564 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068567 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068569 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068573 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068577 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068580 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068583 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068586 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:12:04.080230 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068588 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068591 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068593 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068596 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068598 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068601 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068603 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068606 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068608 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068612 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068616 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068619 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068622 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068625 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068628 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068631 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068634 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068636 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068639 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068644 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:12:04.081038 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068647 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068650 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068653 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068656 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068659 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.068661 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.069165 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.075932 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.075946 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.076733 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.076739 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076794 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076799 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076802 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076805 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:12:04.081631 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076808 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076811 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076814 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076816 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076819 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076824 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076828 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076831 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076834 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076836 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076839 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076842 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076844 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076847 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076850 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076852 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076855 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076858 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076860 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076863 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:12:04.082227 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076865 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076868 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076870 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076873 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076875 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076894 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076898 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076901 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076904 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076907 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076909 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076912 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076915 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076918 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076920 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076923 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076925 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076929 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076933 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:12:04.082712 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076937 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076940 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076942 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076945 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076948 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076951 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076954 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076957 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076960 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076962 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076965 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076968 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076970 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076972 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076975 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076977 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076980 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076984 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076987 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:12:04.083190 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076990 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076993 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076996 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.076999 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077002 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077005 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077007 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077010 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077013 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077015 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077018 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077020 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077023 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077025 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077028 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077031 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077033 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077035 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077038 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077040 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:12:04.083645 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077043 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077045 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077048 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077051 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077053 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077056 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077058 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077061 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077064 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077066 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077068 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077072 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077075 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077077 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077079 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077082 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:12:04.084156 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077085 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.077090 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077198 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077202 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077205 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077208 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077210 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077213 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077215 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077218 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077220 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077223 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077225 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077228 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077230 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:12:04.084527 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077232 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077235 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077237 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077240 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077242 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077245 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077247 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077250 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077252 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077255 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077257 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077259 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077263 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077267 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077270 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077274 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077278 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077281 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077284 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:12:04.084913 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077286 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077289 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077291 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077294 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077296 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077299 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077301 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077304 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077306 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077308 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077311 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077313 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077316 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077318 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077321 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077323 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077325 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077328 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.077358 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078395 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:12:04.085368 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078409 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078416 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078422 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078427 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078432 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078437 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078442 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078447 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078451 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078456 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078461 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078471 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078475 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078480 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078484 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078489 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078493 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078497 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078501 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:12:04.085849 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078506 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078511 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078516 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078520 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078525 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078533 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078538 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078543 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078547 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078552 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078556 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078563 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078571 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078577 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078582 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078586 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078591 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078600 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078605 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:12:04.086322 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078609 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078614 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078618 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078622 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078627 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078631 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078636 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078640 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:04.078645 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.078653 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.078903 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.081745 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.083138 2578 server.go:1019] "Starting client certificate rotation" Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.083242 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:12:04.086778 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.083278 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:12:04.087147 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.085211 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:12:04.087147 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.086270 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:12:04.095709 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.095694 2578 log.go:25] "Validated CRI v1 runtime API" Jul 22 16:12:04.099119 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.099104 2578 log.go:25] "Validated CRI v1 image API" Jul 22 16:12:04.099923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.099900 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 16:12:04.103240 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.103219 2578 fs.go:135] Filesystem UUIDs: map[34156d07-ecef-430a-9c2b-93f9b86f5330:/dev/nvme0n1p3 3ba99901-6089-4d85-aaa4-7b26e6f88dab:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 22 16:12:04.103300 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.103237 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 16:12:04.108658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.108551 2578 manager.go:216] Machine: {Timestamp:2026-07-22 16:12:04.107480754 +0000 UTC m=+0.280005620 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097442 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29642f5ea03a04f8350b1ccd281410 SystemUUID:ec29642f-5ea0-3a04-f835-0b1ccd281410 BootID:00ba25d8-790b-49bf-a6a4-82c62f16754a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3e:7c:c3:e2:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3e:7c:c3:e2:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:d9:81:e2:c6:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 16:12:04.108658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.108653 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 16:12:04.108771 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.108721 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 16:12:04.109518 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.109497 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 16:12:04.109643 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.109520 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-1.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 16:12:04.110110 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.110099 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 16:12:04.110110 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.110111 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 16:12:04.110192 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.110118 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:12:04.110657 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.110647 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:12:04.111545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.111536 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 16:12:04.111908 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.111893 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:12:04.113951 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.113939 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:12:04.114060 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114050 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 16:12:04.114093 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114078 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 22 16:12:04.114093 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114090 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 16:12:04.114166 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114104 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 16:12:04.114166 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114113 2578 kubelet.go:398] "Adding apiserver pod source" Jul 22 16:12:04.114166 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114122 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 16:12:04.114494 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114473 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:12:04.114860 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.114851 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:12:04.118137 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.118115 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 16:12:04.119434 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.119418 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 16:12:04.119491 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.119449 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 16:12:04.120128 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120107 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 16:12:04.120128 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120126 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 16:12:04.120128 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120132 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120138 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120148 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120157 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120168 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120173 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120180 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120186 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120205 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 16:12:04.120263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.120212 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 16:12:04.121517 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.121506 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 16:12:04.121517 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.121516 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 16:12:04.124312 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.124297 2578 server.go:1268] "Started kubelet" Jul 22 16:12:04.124381 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.124345 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 16:12:04.124433 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.124376 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 16:12:04.124479 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.124443 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 16:12:04.125117 ip-10-0-142-1 systemd[1]: Started Kubernetes Kubelet. Jul 22 16:12:04.125489 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.125470 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 16:12:04.126040 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.125996 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-1.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 16:12:04.126139 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.126033 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 16:12:04.126350 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.126250 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 16:12:04.126554 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.126527 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 22 16:12:04.129766 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.129750 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 16:12:04.130470 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.130452 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 16:12:04.130470 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.130469 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 16:12:04.130607 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.130515 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:04.131010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.130902 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 16:12:04.131010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.130907 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 16:12:04.131010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.130926 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 16:12:04.131203 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.131138 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 22 16:12:04.131203 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.131148 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 22 16:12:04.132585 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.132567 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 16:12:04.132585 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.132586 2578 factory.go:55] Registering systemd factory Jul 22 16:12:04.132704 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.132595 2578 factory.go:223] Registration of the systemd container factory successfully Jul 22 16:12:04.133018 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.132968 2578 factory.go:153] Registering CRI-O factory Jul 22 16:12:04.133018 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.132987 2578 factory.go:223] Registration of the crio container factory successfully Jul 22 16:12:04.133018 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.133018 2578 factory.go:103] Registering Raw factory Jul 22 16:12:04.133203 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.133032 2578 manager.go:1196] Started watching for new ooms in manager Jul 22 16:12:04.133463 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.133449 2578 manager.go:349] Starting recovery of all containers Jul 22 16:12:04.135318 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.135286 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 16:12:04.135318 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.135296 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 16:12:04.136528 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.135653 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-1.ec2.internal.18c4a8983b058af9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-1.ec2.internal,UID:ip-10-0-142-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-1.ec2.internal,},FirstTimestamp:2026-07-22 16:12:04.124273401 +0000 UTC m=+0.296798247,LastTimestamp:2026-07-22 16:12:04.124273401 +0000 UTC m=+0.296798247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-1.ec2.internal,}" Jul 22 16:12:04.144905 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.144324 2578 manager.go:354] Recovery completed Jul 22 16:12:04.149331 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.149317 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:12:04.151075 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.151044 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:12:04.151075 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.151074 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:12:04.151172 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.151087 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:12:04.151583 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.151570 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 16:12:04.151583 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.151582 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 16:12:04.151691 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.151596 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:12:04.151721 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.151581 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:12:04.152416 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.152402 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:12:04.152475 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.152427 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:12:04.152475 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.152436 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:12:04.152820 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.152801 2578 policy_none.go:49] "None policy: Start" Jul 22 16:12:04.152899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.152826 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 16:12:04.152899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.152842 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 16:12:04.153255 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.153170 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-1.ec2.internal.18c4a8983c9e47fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-1.ec2.internal,UID:ip-10-0-142-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-1.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-1.ec2.internal,},FirstTimestamp:2026-07-22 16:12:04.151060477 +0000 UTC m=+0.323585322,LastTimestamp:2026-07-22 16:12:04.151060477 +0000 UTC m=+0.323585322,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-1.ec2.internal,}" Jul 22 16:12:04.153907 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.153895 2578 policy_none.go:47] "Start" Jul 22 16:12:04.154499 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.154484 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jfng" Jul 22 16:12:04.164948 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.164876 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-1.ec2.internal.18c4a8983c9e93c5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-1.ec2.internal,UID:ip-10-0-142-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-1.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-1.ec2.internal,},FirstTimestamp:2026-07-22 16:12:04.151079877 +0000 UTC m=+0.323604724,LastTimestamp:2026-07-22 16:12:04.151079877 +0000 UTC m=+0.323604724,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-1.ec2.internal,}" Jul 22 16:12:04.169131 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.169113 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jfng" Jul 22 16:12:04.192479 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.192370 2578 manager.go:337] "Starting Device Plugin manager" Jul 22 16:12:04.192556 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.192496 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 16:12:04.192556 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.192509 2578 server.go:87] "Starting device plugin registration server" Jul 22 16:12:04.192684 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.192674 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 16:12:04.192716 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.192684 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 16:12:04.192812 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.192794 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 16:12:04.192953 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.192863 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 16:12:04.192953 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.192871 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 16:12:04.193277 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.193262 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 16:12:04.193341 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:04.193298 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:04.293208 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.293179 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:12:04.294271 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.293842 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:12:04.294271 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.293864 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:12:04.294271 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.293876 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:12:04.294271 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:04.293926 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.483683 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.483663 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 16:12:05.484962 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.484940 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 16:12:05.485032 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.484984 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 16:12:05.485032 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.485004 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 16:12:05.485112 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.485038 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 16:12:05.487201 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.487182 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:12:05.492249 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.492234 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.492333 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.492253 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-1.ec2.internal\": node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:05.520266 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.520225 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:05.585277 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.585255 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal"] Jul 22 16:12:05.585350 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.585325 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:12:05.586096 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.586082 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:12:05.586167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.586108 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:12:05.586167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.586120 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:12:05.587538 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.587513 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:12:05.587633 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.587620 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.587669 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.587658 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:12:05.588192 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.588169 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:12:05.588192 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.588195 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:12:05.588301 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.588204 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:12:05.588301 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.588174 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:12:05.588301 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.588269 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:12:05.588301 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.588282 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:12:05.589649 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.589632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.589722 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.589655 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:12:05.590303 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.590288 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:12:05.590383 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.590315 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:12:05.590383 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.590324 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:12:05.619087 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.619072 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-1.ec2.internal\" not found" node="ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.619964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.619946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d81f0a8852c1b51dc0d683c3eb97779a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-1.ec2.internal\" (UID: \"d81f0a8852c1b51dc0d683c3eb97779a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.620044 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.619975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea6724a16869ee2376effcabe611a70e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal\" (UID: \"ea6724a16869ee2376effcabe611a70e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.620044 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.620001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea6724a16869ee2376effcabe611a70e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal\" (UID: \"ea6724a16869ee2376effcabe611a70e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.620493 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.620482 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:05.623386 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.623370 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-1.ec2.internal\" not found" node="ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.720956 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.720937 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:05.721040 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.720956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea6724a16869ee2376effcabe611a70e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal\" (UID: \"ea6724a16869ee2376effcabe611a70e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.721040 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.720978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea6724a16869ee2376effcabe611a70e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal\" (UID: \"ea6724a16869ee2376effcabe611a70e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.721040 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.720994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d81f0a8852c1b51dc0d683c3eb97779a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-1.ec2.internal\" (UID: \"d81f0a8852c1b51dc0d683c3eb97779a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.721134 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.721037 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d81f0a8852c1b51dc0d683c3eb97779a-config\") pod \"kube-apiserver-proxy-ip-10-0-142-1.ec2.internal\" (UID: \"d81f0a8852c1b51dc0d683c3eb97779a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.721134 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.721059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ea6724a16869ee2376effcabe611a70e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal\" (UID: \"ea6724a16869ee2376effcabe611a70e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.721134 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.721051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea6724a16869ee2376effcabe611a70e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal\" (UID: \"ea6724a16869ee2376effcabe611a70e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.821308 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.821261 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:05.921188 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.921169 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:05.922250 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:05.922235 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:05.926595 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:05.926575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" Jul 22 16:12:06.023101 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.023077 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:06.123496 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.123444 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:06.223804 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.223784 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-1.ec2.internal\" not found" Jul 22 16:12:06.240452 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.240436 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:12:06.243600 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.243586 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:12:06.270598 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.270580 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 16:12:06.270932 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.270683 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:12:06.270932 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.270686 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:12:06.270932 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.270683 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:12:06.302866 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.302848 2578 apiserver.go:52] "Watching apiserver" Jul 22 16:12:06.318639 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.318620 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" Jul 22 16:12:06.318731 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.318653 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 16:12:06.323679 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.323663 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 16:12:06.324392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.324235 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mwwp5","openshift-ovn-kubernetes/ovnkube-node-87d4v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf","openshift-dns/node-resolver-4hj59","openshift-image-registry/node-ca-wcksj","openshift-multus/multus-additional-cni-plugins-5rns6","openshift-network-diagnostics/network-check-target-gpxss","openshift-network-operator/iptables-alerter-782hj","kube-system/global-pull-secret-syncer-z8drf","kube-system/konnectivity-agent-lhwcp","openshift-cluster-node-tuning-operator/tuned-g8s8h","openshift-multus/multus-t46jb"] Jul 22 16:12:06.330762 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.330738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:06.330888 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.330825 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:06.332658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.332637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.333386 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.333364 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:12:06.333509 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.333452 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" Jul 22 16:12:06.333709 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.333691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.334893 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.334874 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.335372 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.335353 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 16:12:06.335675 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.335659 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 16:12:06.336217 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336200 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 16:12:06.336380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336236 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 16:12:06.336380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336246 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 16:12:06.336380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336275 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 16:12:06.336380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336280 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-flklf\"" Jul 22 16:12:06.336567 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336552 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 16:12:06.336633 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.336633 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.336585 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 16:12:06.337048 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.337024 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6779r\"" Jul 22 16:12:06.337129 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.337099 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 16:12:06.337440 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.337424 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pnprs\"" Jul 22 16:12:06.337440 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.337435 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 16:12:06.337510 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.337463 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 16:12:06.338016 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.338001 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.338771 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.338754 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:12:06.339150 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.339124 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 16:12:06.339236 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.339212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:06.339309 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.339287 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:06.339351 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.339301 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 16:12:06.339405 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.339392 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhzrz\"" Jul 22 16:12:06.339483 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.339470 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 16:12:06.339985 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.339972 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 16:12:06.340316 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.340301 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 16:12:06.340473 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.340456 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 16:12:06.340511 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.340466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.340511 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.340477 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lbpfp\"" Jul 22 16:12:06.340703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.340687 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 16:12:06.340764 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.340690 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 16:12:06.341746 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.341732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.341804 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.341787 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:06.342742 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.342725 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.342915 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.342892 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 16:12:06.343087 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.343074 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:12:06.343202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.343188 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bvfq9\"" Jul 22 16:12:06.343246 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.343193 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 16:12:06.343355 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.343342 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:12:06.344049 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.344035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.345262 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.345244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.345262 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.345259 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal"] Jul 22 16:12:06.345364 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.345344 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 16:12:06.345402 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.345378 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal"] Jul 22 16:12:06.346569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.346283 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pnqcz\"" Jul 22 16:12:06.346569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.346343 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 16:12:06.346791 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.346772 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:12:06.347109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.347088 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 16:12:06.347202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.347167 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7q59s\"" Jul 22 16:12:06.347536 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.347509 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s7x77\"" Jul 22 16:12:06.347807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.347792 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 16:12:06.360706 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.360673 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 16:07:05 +0000 UTC" deadline="2027-12-24 21:12:32.029276847 +0000 UTC" Jul 22 16:12:06.360706 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.360704 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12485h0m25.668575334s" Jul 22 16:12:06.376067 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.376023 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cd5m7" Jul 22 16:12:06.390425 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.389968 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cd5m7" Jul 22 16:12:06.419485 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.419467 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 16:12:06.425028 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-device-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.425101 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysconfig\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.425101 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysctl-conf\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.425200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-run\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.425200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46hp\" (UniqueName: \"kubernetes.io/projected/f420df98-88df-4d05-97a6-47076aa5450e-kube-api-access-c46hp\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.425283 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-hosts-file\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.425283 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-cnibin\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.425351 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv8sn\" (UniqueName: \"kubernetes.io/projected/06184455-9451-417f-9a46-2009651babfc-kube-api-access-zv8sn\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:06.425351 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-etc-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.425351 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovnkube-config\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.425578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-registration-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.425578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.425578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.425578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-k8s-cni-cncf-io\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.425578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-conf-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.425578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-kubelet\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.425578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-systemd\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-node-log\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-host\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-kubernetes\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/937727a3-12c8-486f-b385-a19d18eef8dd-cni-binary-copy\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-etc-kubernetes\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-slash\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-hostroot\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-sys\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gwq\" (UniqueName: \"kubernetes.io/projected/937727a3-12c8-486f-b385-a19d18eef8dd-kube-api-access-z7gwq\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.425815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-host\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-host-slash\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-system-cni-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-netns\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-kubelet\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lp8\" (UniqueName: \"kubernetes.io/projected/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-kube-api-access-k2lp8\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-iptables-alerter-script\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.425996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-tmp-dir\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426018 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-sys-fs\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqr5\" (UniqueName: \"kubernetes.io/projected/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-kube-api-access-nrqr5\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysctl-d\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-cni-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-systemd\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.426360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-cni-bin\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-run-netns\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-cni-bin\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c68118a-8c93-478a-9780-3852ef7091ab-kubelet-config\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-lib-modules\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/532eab52-496c-4fc8-aada-0282a49edc3c-konnectivity-ca\") pod \"konnectivity-agent-lhwcp\" (UID: \"532eab52-496c-4fc8-aada-0282a49edc3c\") " pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-cni-multus\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-multus-certs\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-var-lib-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-socket-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmff2\" (UniqueName: \"kubernetes.io/projected/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-kube-api-access-vmff2\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-modprobe-d\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-var-lib-kubelet\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-socket-dir-parent\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.426996 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-ovn\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-systemd-units\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-cnibin\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwwr\" (UniqueName: \"kubernetes.io/projected/a2076767-84a2-4ba2-b556-30c266b278b6-kube-api-access-hrwwr\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7rk\" (UniqueName: \"kubernetes.io/projected/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-kube-api-access-kz7rk\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/532eab52-496c-4fc8-aada-0282a49edc3c-agent-certs\") pod \"konnectivity-agent-lhwcp\" (UID: \"532eab52-496c-4fc8-aada-0282a49edc3c\") " pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-cni-netd\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/937727a3-12c8-486f-b385-a19d18eef8dd-multus-daemon-config\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-run-ovn-kubernetes\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-env-overrides\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovn-node-metrics-cert\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovnkube-script-lib\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-serviceca\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c68118a-8c93-478a-9780-3852ef7091ab-dbus\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.427508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f420df98-88df-4d05-97a6-47076aa5450e-etc-tuned\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.427992 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.426992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-log-socket\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427992 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.427015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42c4v\" (UniqueName: \"kubernetes.io/projected/384883b4-4ca8-4839-8607-4f9536c3ebf2-kube-api-access-42c4v\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.427992 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.427034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f420df98-88df-4d05-97a6-47076aa5450e-tmp\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.427992 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.427050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-os-release\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.427992 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.427083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-os-release\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.433782 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.433748 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81f0a8852c1b51dc0d683c3eb97779a.slice/crio-1e99c5fa26474723980017676e2a94012b91b7d3dd509f7eb8404c84a4a2615b WatchSource:0}: Error finding container 1e99c5fa26474723980017676e2a94012b91b7d3dd509f7eb8404c84a4a2615b: Status 404 returned error can't find the container with id 1e99c5fa26474723980017676e2a94012b91b7d3dd509f7eb8404c84a4a2615b Jul 22 16:12:06.434442 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.434429 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6724a16869ee2376effcabe611a70e.slice/crio-db8ed50e15387351d6bfa4b6bb06888370c6e9ef3b1248115fee488bd2c9f3a2 WatchSource:0}: Error finding container db8ed50e15387351d6bfa4b6bb06888370c6e9ef3b1248115fee488bd2c9f3a2: Status 404 returned error can't find the container with id db8ed50e15387351d6bfa4b6bb06888370c6e9ef3b1248115fee488bd2c9f3a2 Jul 22 16:12:06.440914 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.440890 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:12:06.488968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.488933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" event={"ID":"ea6724a16869ee2376effcabe611a70e","Type":"ContainerStarted","Data":"db8ed50e15387351d6bfa4b6bb06888370c6e9ef3b1248115fee488bd2c9f3a2"} Jul 22 16:12:06.489916 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.489886 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" event={"ID":"d81f0a8852c1b51dc0d683c3eb97779a","Type":"ContainerStarted","Data":"1e99c5fa26474723980017676e2a94012b91b7d3dd509f7eb8404c84a4a2615b"} Jul 22 16:12:06.527781 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-log-socket\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.527835 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42c4v\" (UniqueName: \"kubernetes.io/projected/384883b4-4ca8-4839-8607-4f9536c3ebf2-kube-api-access-42c4v\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.527835 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f420df98-88df-4d05-97a6-47076aa5450e-tmp\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.527835 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-os-release\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.527923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-os-release\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.527923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-log-socket\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.527923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527905 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-os-release\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.527923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527908 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-os-release\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.528054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-device-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.528054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysconfig\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.528054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.527989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysctl-conf\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.528054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-run\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.528054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46hp\" (UniqueName: \"kubernetes.io/projected/f420df98-88df-4d05-97a6-47076aa5450e-kube-api-access-c46hp\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.528054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-device-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.528054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysconfig\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-hosts-file\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528072 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528141 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysctl-conf\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-cnibin\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv8sn\" (UniqueName: \"kubernetes.io/projected/06184455-9451-417f-9a46-2009651babfc-kube-api-access-zv8sn\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-hosts-file\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528221 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-run\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-etc-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528253 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-cnibin\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovnkube-config\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-registration-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528306 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-etc-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-k8s-cni-cncf-io\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.528388 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528359 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-registration-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-k8s-cni-cncf-io\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-conf-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-kubelet\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-systemd\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528484 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-conf-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528494 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-node-log\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-host\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-kubelet\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-kubernetes\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528580 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-systemd\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-kubernetes\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528602 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-host\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-node-log\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/937727a3-12c8-486f-b385-a19d18eef8dd-cni-binary-copy\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-etc-kubernetes\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-slash\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-etc-kubernetes\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-slash\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-hostroot\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528818 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovnkube-config\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-sys\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gwq\" (UniqueName: \"kubernetes.io/projected/937727a3-12c8-486f-b385-a19d18eef8dd-kube-api-access-z7gwq\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-host\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528896 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-sys\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-host-slash\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528921 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-system-cni-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528937 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-hostroot\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-netns\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528981 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-system-cni-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.529927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528986 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-host\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.528990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-kubelet\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-netns\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529025 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-kubelet\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-host-slash\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lp8\" (UniqueName: \"kubernetes.io/projected/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-kube-api-access-k2lp8\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-iptables-alerter-script\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-tmp-dir\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529154 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/937727a3-12c8-486f-b385-a19d18eef8dd-cni-binary-copy\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-sys-fs\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529195 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqr5\" (UniqueName: \"kubernetes.io/projected/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-kube-api-access-nrqr5\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysctl-d\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529239 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-sys-fs\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-cni-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529345 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-sysctl-d\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-systemd\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-cni-bin\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-run-netns\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-tmp-dir\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-cni-bin\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.529431 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-cni-dir\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-systemd\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529466 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-cni-bin\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.529491 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:12:07.029471124 +0000 UTC m=+2.014355127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529495 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-cni-bin\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-run-netns\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c68118a-8c93-478a-9780-3852ef7091ab-kubelet-config\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.531606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-lib-modules\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529598 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/532eab52-496c-4fc8-aada-0282a49edc3c-konnectivity-ca\") pod \"konnectivity-agent-lhwcp\" (UID: \"532eab52-496c-4fc8-aada-0282a49edc3c\") " pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c68118a-8c93-478a-9780-3852ef7091ab-kubelet-config\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-cni-multus\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529604 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-iptables-alerter-script\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-multus-certs\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-var-lib-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529707 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-var-lib-cni-multus\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-host-run-multus-certs\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529741 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-var-lib-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529701 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-lib-modules\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-socket-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529841 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-openvswitch\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmff2\" (UniqueName: \"kubernetes.io/projected/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-kube-api-access-vmff2\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-modprobe-d\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.532121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-var-lib-kubelet\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-socket-dir-parent\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-ovn\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-systemd-units\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529984 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-etc-modprobe-d\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.529996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530010 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-socket-dir\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-cnibin\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530032 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-run-ovn\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/937727a3-12c8-486f-b385-a19d18eef8dd-multus-socket-dir-parent\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530065 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2076767-84a2-4ba2-b556-30c266b278b6-cnibin\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-systemd-units\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530088 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f420df98-88df-4d05-97a6-47076aa5450e-var-lib-kubelet\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530095 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/532eab52-496c-4fc8-aada-0282a49edc3c-konnectivity-ca\") pod \"konnectivity-agent-lhwcp\" (UID: \"532eab52-496c-4fc8-aada-0282a49edc3c\") " pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.530111 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.530158 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret podName:1c68118a-8c93-478a-9780-3852ef7091ab nodeName:}" failed. No retries permitted until 2026-07-22 16:12:07.030141387 +0000 UTC m=+2.015025390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret") pod "global-pull-secret-syncer-z8drf" (UID: "1c68118a-8c93-478a-9780-3852ef7091ab") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:06.532617 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwwr\" (UniqueName: \"kubernetes.io/projected/a2076767-84a2-4ba2-b556-30c266b278b6-kube-api-access-hrwwr\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7rk\" (UniqueName: \"kubernetes.io/projected/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-kube-api-access-kz7rk\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/532eab52-496c-4fc8-aada-0282a49edc3c-agent-certs\") pod \"konnectivity-agent-lhwcp\" (UID: \"532eab52-496c-4fc8-aada-0282a49edc3c\") " pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-cni-netd\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/937727a3-12c8-486f-b385-a19d18eef8dd-multus-daemon-config\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-run-ovn-kubernetes\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-env-overrides\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovn-node-metrics-cert\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530408 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-cni-netd\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovnkube-script-lib\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-serviceca\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530746 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/384883b4-4ca8-4839-8607-4f9536c3ebf2-host-run-ovn-kubernetes\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530801 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2076767-84a2-4ba2-b556-30c266b278b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c68118a-8c93-478a-9780-3852ef7091ab-dbus\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f420df98-88df-4d05-97a6-47076aa5450e-etc-tuned\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovnkube-script-lib\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533202 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.530960 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c68118a-8c93-478a-9780-3852ef7091ab-dbus\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:06.533711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.531016 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/937727a3-12c8-486f-b385-a19d18eef8dd-multus-daemon-config\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.533711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.531326 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/384883b4-4ca8-4839-8607-4f9536c3ebf2-env-overrides\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.531369 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-serviceca\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.533711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.531727 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f420df98-88df-4d05-97a6-47076aa5450e-tmp\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.533711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.532841 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f420df98-88df-4d05-97a6-47076aa5450e-etc-tuned\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.533711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.532856 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/384883b4-4ca8-4839-8607-4f9536c3ebf2-ovn-node-metrics-cert\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.533711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.532876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/532eab52-496c-4fc8-aada-0282a49edc3c-agent-certs\") pod \"konnectivity-agent-lhwcp\" (UID: \"532eab52-496c-4fc8-aada-0282a49edc3c\") " pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.559601 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.559572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmff2\" (UniqueName: \"kubernetes.io/projected/dc2ff436-4fc2-4f0f-882c-e290d93b85c8-kube-api-access-vmff2\") pod \"aws-ebs-csi-driver-node-l6qpf\" (UID: \"dc2ff436-4fc2-4f0f-882c-e290d93b85c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.559702 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.559672 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lp8\" (UniqueName: \"kubernetes.io/projected/0a3df486-be43-4d53-ac36-3e6ad9cfcb80-kube-api-access-k2lp8\") pod \"node-resolver-4hj59\" (UID: \"0a3df486-be43-4d53-ac36-3e6ad9cfcb80\") " pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.560059 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.560040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gwq\" (UniqueName: \"kubernetes.io/projected/937727a3-12c8-486f-b385-a19d18eef8dd-kube-api-access-z7gwq\") pod \"multus-t46jb\" (UID: \"937727a3-12c8-486f-b385-a19d18eef8dd\") " pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.560059 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.560050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv8sn\" (UniqueName: \"kubernetes.io/projected/06184455-9451-417f-9a46-2009651babfc-kube-api-access-zv8sn\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:06.560756 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.560739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwwr\" (UniqueName: \"kubernetes.io/projected/a2076767-84a2-4ba2-b556-30c266b278b6-kube-api-access-hrwwr\") pod \"multus-additional-cni-plugins-5rns6\" (UID: \"a2076767-84a2-4ba2-b556-30c266b278b6\") " pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.560850 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.560835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46hp\" (UniqueName: \"kubernetes.io/projected/f420df98-88df-4d05-97a6-47076aa5450e-kube-api-access-c46hp\") pod \"tuned-g8s8h\" (UID: \"f420df98-88df-4d05-97a6-47076aa5450e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.563151 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.563133 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:12:06.563151 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.563151 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:12:06.563279 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.563164 2578 projected.go:196] Error preparing data for projected volume kube-api-access-64r7p for pod openshift-network-diagnostics/network-check-target-gpxss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:06.563279 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:06.563222 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p podName:9969950d-f2c9-424c-9b21-1e6ffd76789d nodeName:}" failed. No retries permitted until 2026-07-22 16:12:07.063196894 +0000 UTC m=+2.048080879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-64r7p" (UniqueName: "kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p") pod "network-check-target-gpxss" (UID: "9969950d-f2c9-424c-9b21-1e6ffd76789d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:06.563964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.563946 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42c4v\" (UniqueName: \"kubernetes.io/projected/384883b4-4ca8-4839-8607-4f9536c3ebf2-kube-api-access-42c4v\") pod \"ovnkube-node-87d4v\" (UID: \"384883b4-4ca8-4839-8607-4f9536c3ebf2\") " pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.564878 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.564860 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7rk\" (UniqueName: \"kubernetes.io/projected/87a162c9-a265-4c07-8948-ed9a6dc1c8cf-kube-api-access-kz7rk\") pod \"iptables-alerter-782hj\" (UID: \"87a162c9-a265-4c07-8948-ed9a6dc1c8cf\") " pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.564945 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.564908 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqr5\" (UniqueName: \"kubernetes.io/projected/1a83e2e3-3411-44ab-bd64-6fefd51c5a3c-kube-api-access-nrqr5\") pod \"node-ca-wcksj\" (UID: \"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c\") " pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.661295 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.661243 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:06.668821 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.668797 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384883b4_4ca8_4839_8607_4f9536c3ebf2.slice/crio-964feec192b2b8d3ff2f48ff7808f6a6bad88fd31001531d18a2a3cc719f07d7 WatchSource:0}: Error finding container 964feec192b2b8d3ff2f48ff7808f6a6bad88fd31001531d18a2a3cc719f07d7: Status 404 returned error can't find the container with id 964feec192b2b8d3ff2f48ff7808f6a6bad88fd31001531d18a2a3cc719f07d7 Jul 22 16:12:06.683359 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.683344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" Jul 22 16:12:06.689483 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.689464 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2ff436_4fc2_4f0f_882c_e290d93b85c8.slice/crio-720b4ab122f370d90ffb64175f0c82d245d9b40d824eb4a6292c9dff753221a2 WatchSource:0}: Error finding container 720b4ab122f370d90ffb64175f0c82d245d9b40d824eb4a6292c9dff753221a2: Status 404 returned error can't find the container with id 720b4ab122f370d90ffb64175f0c82d245d9b40d824eb4a6292c9dff753221a2 Jul 22 16:12:06.695209 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.695192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hj59" Jul 22 16:12:06.701814 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.701792 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3df486_be43_4d53_ac36_3e6ad9cfcb80.slice/crio-84c7d63760c1ef6517ba9a86571e94559f4e391d13a07fa76f51dda964d9b41a WatchSource:0}: Error finding container 84c7d63760c1ef6517ba9a86571e94559f4e391d13a07fa76f51dda964d9b41a: Status 404 returned error can't find the container with id 84c7d63760c1ef6517ba9a86571e94559f4e391d13a07fa76f51dda964d9b41a Jul 22 16:12:06.714367 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.714351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wcksj" Jul 22 16:12:06.720826 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.720808 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a83e2e3_3411_44ab_bd64_6fefd51c5a3c.slice/crio-60ad8e13d42bce803c36e3ceb384033699c86deb7218fb085cb31bc635c92de9 WatchSource:0}: Error finding container 60ad8e13d42bce803c36e3ceb384033699c86deb7218fb085cb31bc635c92de9: Status 404 returned error can't find the container with id 60ad8e13d42bce803c36e3ceb384033699c86deb7218fb085cb31bc635c92de9 Jul 22 16:12:06.730010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.729993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rns6" Jul 22 16:12:06.741622 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.741589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-782hj" Jul 22 16:12:06.746707 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.746691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:06.749819 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.749795 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a162c9_a265_4c07_8948_ed9a6dc1c8cf.slice/crio-e84fa37616c48ebf41f9134ee43bbf5d59f749781bc37d62f3a67d100101908e WatchSource:0}: Error finding container e84fa37616c48ebf41f9134ee43bbf5d59f749781bc37d62f3a67d100101908e: Status 404 returned error can't find the container with id e84fa37616c48ebf41f9134ee43bbf5d59f749781bc37d62f3a67d100101908e Jul 22 16:12:06.752861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.752845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" Jul 22 16:12:06.753927 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.753909 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532eab52_496c_4fc8_aada_0282a49edc3c.slice/crio-e7d4df88505e9196d2e2c6000dbba20c4b89ecfa3064c3e39324e6854b928911 WatchSource:0}: Error finding container e7d4df88505e9196d2e2c6000dbba20c4b89ecfa3064c3e39324e6854b928911: Status 404 returned error can't find the container with id e7d4df88505e9196d2e2c6000dbba20c4b89ecfa3064c3e39324e6854b928911 Jul 22 16:12:06.757248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.757202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t46jb" Jul 22 16:12:06.760006 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.759979 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf420df98_88df_4d05_97a6_47076aa5450e.slice/crio-d097c03253ff1cfca122b3602d1768e51c226da967e129a0524c62eff6680fa2 WatchSource:0}: Error finding container d097c03253ff1cfca122b3602d1768e51c226da967e129a0524c62eff6680fa2: Status 404 returned error can't find the container with id d097c03253ff1cfca122b3602d1768e51c226da967e129a0524c62eff6680fa2 Jul 22 16:12:06.760997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:06.760979 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:12:06.764897 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:06.764876 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937727a3_12c8_486f_b385_a19d18eef8dd.slice/crio-6c587e23aec87f97deea5acef4deaeaad243ae674a7615d55af987741eda8107 WatchSource:0}: Error finding container 6c587e23aec87f97deea5acef4deaeaad243ae674a7615d55af987741eda8107: Status 404 returned error can't find the container with id 6c587e23aec87f97deea5acef4deaeaad243ae674a7615d55af987741eda8107 Jul 22 16:12:07.035755 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.035677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:07.035755 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.035741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:07.035960 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.035825 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:07.035960 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.035835 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:07.035960 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.035892 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret podName:1c68118a-8c93-478a-9780-3852ef7091ab nodeName:}" failed. No retries permitted until 2026-07-22 16:12:08.035870397 +0000 UTC m=+3.020754394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret") pod "global-pull-secret-syncer-z8drf" (UID: "1c68118a-8c93-478a-9780-3852ef7091ab") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:07.035960 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.035917 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:12:08.035906199 +0000 UTC m=+3.020790186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:07.137130 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.137096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:07.137296 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.137218 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:12:07.137296 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.137238 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:12:07.137296 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.137250 2578 projected.go:196] Error preparing data for projected volume kube-api-access-64r7p for pod openshift-network-diagnostics/network-check-target-gpxss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:07.137450 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.137311 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p podName:9969950d-f2c9-424c-9b21-1e6ffd76789d nodeName:}" failed. No retries permitted until 2026-07-22 16:12:08.137293683 +0000 UTC m=+3.122177673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-64r7p" (UniqueName: "kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p") pod "network-check-target-gpxss" (UID: "9969950d-f2c9-424c-9b21-1e6ffd76789d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:07.392891 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.392811 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:07:06 +0000 UTC" deadline="2028-02-20 06:14:43.876799291 +0000 UTC" Jul 22 16:12:07.392891 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.392849 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13862h2m36.483954672s" Jul 22 16:12:07.487562 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.487535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:07.487721 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.487670 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:07.488051 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.488031 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:07.488145 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:07.488125 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:07.499016 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.498986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lhwcp" event={"ID":"532eab52-496c-4fc8-aada-0282a49edc3c","Type":"ContainerStarted","Data":"e7d4df88505e9196d2e2c6000dbba20c4b89ecfa3064c3e39324e6854b928911"} Jul 22 16:12:07.504942 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.504913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-782hj" event={"ID":"87a162c9-a265-4c07-8948-ed9a6dc1c8cf","Type":"ContainerStarted","Data":"e84fa37616c48ebf41f9134ee43bbf5d59f749781bc37d62f3a67d100101908e"} Jul 22 16:12:07.506724 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.506698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerStarted","Data":"f4dd05e9af28f7ff4e1434af2af6631fad6c1395140a662f72a4e3225875ee0e"} Jul 22 16:12:07.514441 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.514417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hj59" event={"ID":"0a3df486-be43-4d53-ac36-3e6ad9cfcb80","Type":"ContainerStarted","Data":"84c7d63760c1ef6517ba9a86571e94559f4e391d13a07fa76f51dda964d9b41a"} Jul 22 16:12:07.524359 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.524246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"964feec192b2b8d3ff2f48ff7808f6a6bad88fd31001531d18a2a3cc719f07d7"} Jul 22 16:12:07.528429 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.528404 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t46jb" event={"ID":"937727a3-12c8-486f-b385-a19d18eef8dd","Type":"ContainerStarted","Data":"6c587e23aec87f97deea5acef4deaeaad243ae674a7615d55af987741eda8107"} Jul 22 16:12:07.535213 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.535187 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" event={"ID":"f420df98-88df-4d05-97a6-47076aa5450e","Type":"ContainerStarted","Data":"d097c03253ff1cfca122b3602d1768e51c226da967e129a0524c62eff6680fa2"} Jul 22 16:12:07.537859 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.537817 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wcksj" event={"ID":"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c","Type":"ContainerStarted","Data":"60ad8e13d42bce803c36e3ceb384033699c86deb7218fb085cb31bc635c92de9"} Jul 22 16:12:07.541264 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.541240 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" event={"ID":"dc2ff436-4fc2-4f0f-882c-e290d93b85c8","Type":"ContainerStarted","Data":"720b4ab122f370d90ffb64175f0c82d245d9b40d824eb4a6292c9dff753221a2"} Jul 22 16:12:07.664002 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.663923 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:12:07.909611 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:07.909374 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:12:08.044278 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:08.043506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:08.044278 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:08.043573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:08.044278 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.043727 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:08.044278 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.043783 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret podName:1c68118a-8c93-478a-9780-3852ef7091ab nodeName:}" failed. No retries permitted until 2026-07-22 16:12:10.043765599 +0000 UTC m=+5.028649590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret") pod "global-pull-secret-syncer-z8drf" (UID: "1c68118a-8c93-478a-9780-3852ef7091ab") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:08.044278 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.044156 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:08.044278 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.044201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:12:10.044186924 +0000 UTC m=+5.029070911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:08.144577 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:08.144541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:08.144765 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.144684 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:12:08.144765 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.144713 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:12:08.144765 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.144725 2578 projected.go:196] Error preparing data for projected volume kube-api-access-64r7p for pod openshift-network-diagnostics/network-check-target-gpxss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:08.144923 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.144772 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p podName:9969950d-f2c9-424c-9b21-1e6ffd76789d nodeName:}" failed. No retries permitted until 2026-07-22 16:12:10.144755906 +0000 UTC m=+5.129639905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-64r7p" (UniqueName: "kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p") pod "network-check-target-gpxss" (UID: "9969950d-f2c9-424c-9b21-1e6ffd76789d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:08.393851 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:08.393740 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:07:06 +0000 UTC" deadline="2028-03-27 12:34:20.716485363 +0000 UTC" Jul 22 16:12:08.393851 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:08.393803 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14732h22m12.322687033s" Jul 22 16:12:08.486022 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:08.485935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:08.486184 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:08.486048 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:09.054284 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:09.054253 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:12:09.486209 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:09.485649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:09.486209 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:09.485816 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:09.486209 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:09.485879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:09.486209 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:09.485998 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:10.059760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:10.059725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:10.059950 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:10.059783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:10.059950 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.059862 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:10.059950 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.059882 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:10.059950 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.059932 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:12:14.059910776 +0000 UTC m=+9.044794775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:10.059950 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.059951 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret podName:1c68118a-8c93-478a-9780-3852ef7091ab nodeName:}" failed. No retries permitted until 2026-07-22 16:12:14.059942251 +0000 UTC m=+9.044826238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret") pod "global-pull-secret-syncer-z8drf" (UID: "1c68118a-8c93-478a-9780-3852ef7091ab") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:10.161113 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:10.161078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:10.161274 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.161234 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:12:10.161274 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.161258 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:12:10.161274 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.161270 2578 projected.go:196] Error preparing data for projected volume kube-api-access-64r7p for pod openshift-network-diagnostics/network-check-target-gpxss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:10.161439 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.161324 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p podName:9969950d-f2c9-424c-9b21-1e6ffd76789d nodeName:}" failed. No retries permitted until 2026-07-22 16:12:14.161305441 +0000 UTC m=+9.146189431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-64r7p" (UniqueName: "kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p") pod "network-check-target-gpxss" (UID: "9969950d-f2c9-424c-9b21-1e6ffd76789d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:10.485364 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:10.485281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:10.485550 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:10.485409 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:11.486389 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:11.485604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:11.486389 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:11.485732 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:11.486389 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:11.485792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:11.486389 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:11.485867 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:12.485324 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:12.485265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:12.485486 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:12.485385 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:13.486273 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:13.486228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:13.486705 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:13.486373 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:13.486773 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:13.486720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:13.486856 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:13.486819 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:14.096195 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:14.095426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:14.096195 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:14.095489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:14.096195 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.095668 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:14.096195 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.095730 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret podName:1c68118a-8c93-478a-9780-3852ef7091ab nodeName:}" failed. No retries permitted until 2026-07-22 16:12:22.095711572 +0000 UTC m=+17.080595557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret") pod "global-pull-secret-syncer-z8drf" (UID: "1c68118a-8c93-478a-9780-3852ef7091ab") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:14.096195 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.096115 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:14.096195 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.096159 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:12:22.096145116 +0000 UTC m=+17.081029115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:14.196476 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:14.196438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:14.196664 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.196604 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:12:14.196664 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.196632 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:12:14.196664 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.196644 2578 projected.go:196] Error preparing data for projected volume kube-api-access-64r7p for pod openshift-network-diagnostics/network-check-target-gpxss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:14.196819 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.196709 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p podName:9969950d-f2c9-424c-9b21-1e6ffd76789d nodeName:}" failed. No retries permitted until 2026-07-22 16:12:22.196689274 +0000 UTC m=+17.181573273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-64r7p" (UniqueName: "kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p") pod "network-check-target-gpxss" (UID: "9969950d-f2c9-424c-9b21-1e6ffd76789d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:14.485661 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:14.485587 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:14.485821 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:14.485722 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:15.489321 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:15.489291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:15.489707 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:15.489404 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:15.489707 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:15.489614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:15.489812 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:15.489708 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:16.485398 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:16.485354 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:16.485590 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:16.485480 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:17.488155 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:17.488127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:17.488581 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:17.488131 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:17.488581 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:17.488254 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:17.488581 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:17.488347 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:18.485339 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:18.485301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:18.485493 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:18.485411 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:19.485415 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:19.485372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:19.485871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:19.485372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:19.485871 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:19.485514 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:19.485871 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:19.485626 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:20.486114 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:20.486085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:20.486472 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:20.486186 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:21.485817 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:21.485770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:21.486001 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:21.485894 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:21.486001 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:21.485946 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:21.486106 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:21.486046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:22.160451 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:22.160424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:22.160850 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:22.160481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:22.160850 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.160584 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:22.160850 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.160603 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:22.160850 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.160641 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:12:38.160626734 +0000 UTC m=+33.145510718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:22.160850 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.160655 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret podName:1c68118a-8c93-478a-9780-3852ef7091ab nodeName:}" failed. No retries permitted until 2026-07-22 16:12:38.160648806 +0000 UTC m=+33.145532797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret") pod "global-pull-secret-syncer-z8drf" (UID: "1c68118a-8c93-478a-9780-3852ef7091ab") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:22.261607 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:22.261585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:22.261710 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.261698 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:12:22.261751 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.261714 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:12:22.261751 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.261723 2578 projected.go:196] Error preparing data for projected volume kube-api-access-64r7p for pod openshift-network-diagnostics/network-check-target-gpxss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:22.261814 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.261762 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p podName:9969950d-f2c9-424c-9b21-1e6ffd76789d nodeName:}" failed. No retries permitted until 2026-07-22 16:12:38.261747524 +0000 UTC m=+33.246631509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-64r7p" (UniqueName: "kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p") pod "network-check-target-gpxss" (UID: "9969950d-f2c9-424c-9b21-1e6ffd76789d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:22.485346 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:22.485280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:22.485476 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:22.485393 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:23.488126 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:23.488094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:23.488499 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:23.488202 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:23.488499 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:23.488094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:23.488499 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:23.488377 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:24.485694 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:24.485661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:24.485860 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:24.485764 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:25.486729 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:25.486700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:25.487165 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:25.486837 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:25.487165 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:25.486903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:25.487165 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:25.487003 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:26.485684 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:26.485662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:26.485763 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:26.485752 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:26.587567 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:26.587538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" event={"ID":"f420df98-88df-4d05-97a6-47076aa5450e","Type":"ContainerStarted","Data":"a2b8ad67b7fef9f9c77a3b779eb742fb0b4de80e570382ec4e8c5616d985272b"} Jul 22 16:12:26.588943 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:26.588921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" event={"ID":"d81f0a8852c1b51dc0d683c3eb97779a","Type":"ContainerStarted","Data":"b2c94d3e75fa25c3622707931e8be0779d7eec523c8b1730e631f114e9daf1af"} Jul 22 16:12:26.591612 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:26.591548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"6b5a9cee357e7e1016f2c938d0105e5a2f12e87a46de2e8d55ee487922f86455"} Jul 22 16:12:26.591612 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:26.591608 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"f6ba96f97e059c98f39aa42400db6a79e0dc6e95d5fb2e7064a6cbe67c81c8b2"} Jul 22 16:12:26.626348 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:26.626309 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-1.ec2.internal" podStartSLOduration=20.626297387 podStartE2EDuration="20.626297387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:26.626229939 +0000 UTC m=+21.611113946" watchObservedRunningTime="2026-07-22 16:12:26.626297387 +0000 UTC m=+21.611181393" Jul 22 16:12:26.626498 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:26.626434 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g8s8h" podStartSLOduration=2.13549023 podStartE2EDuration="21.626425404s" totalImagesPullingTime="19.490935174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:26.609393666 +0000 UTC m=+21.594277673" watchObservedRunningTime="2026-07-22 16:12:26.626425404 +0000 UTC m=+21.611309414" Jul 22 16:12:27.486082 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.485924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:27.486207 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.485980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:27.486207 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:27.486169 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:27.486323 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:27.486270 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:27.594970 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.594888 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lhwcp" event={"ID":"532eab52-496c-4fc8-aada-0282a49edc3c","Type":"ContainerStarted","Data":"ef09759346931db677ac4cdd7f85089bcc29d4dfe89ea4f5b3662bf903a411ee"} Jul 22 16:12:27.596629 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.596602 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-782hj" event={"ID":"87a162c9-a265-4c07-8948-ed9a6dc1c8cf","Type":"ContainerStarted","Data":"9faf0a9972051abf29054be0979dbc74a2a0bc7dd848e448223e3569526cf99b"} Jul 22 16:12:27.598304 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.598277 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2076767-84a2-4ba2-b556-30c266b278b6" containerID="d12bc622d5cbad37b4fb3ce399a4d55aa7eceacd9915559f2b966b41f75d5293" exitCode=0 Jul 22 16:12:27.598425 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.598367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerDied","Data":"d12bc622d5cbad37b4fb3ce399a4d55aa7eceacd9915559f2b966b41f75d5293"} Jul 22 16:12:27.600306 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.600283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hj59" event={"ID":"0a3df486-be43-4d53-ac36-3e6ad9cfcb80","Type":"ContainerStarted","Data":"008470eea8c61f8b1bfa6068ba7355cca824c98f1d0a5ccae89ec127811b106d"} Jul 22 16:12:27.603591 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.603565 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:12:27.603965 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.603942 2578 generic.go:356] "Generic (PLEG): container finished" podID="384883b4-4ca8-4839-8607-4f9536c3ebf2" containerID="6b5a9cee357e7e1016f2c938d0105e5a2f12e87a46de2e8d55ee487922f86455" exitCode=1 Jul 22 16:12:27.604057 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.604000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerDied","Data":"6b5a9cee357e7e1016f2c938d0105e5a2f12e87a46de2e8d55ee487922f86455"} Jul 22 16:12:27.604115 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.604080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"4f983309967724f56a02aa4dd75b87071683a367daa125c490d8364aba3e78af"} Jul 22 16:12:27.604176 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.604159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"055a350e13515fdbe667d50f7293d6b97e2394d8b3e31b87018b716bd9ff4420"} Jul 22 16:12:27.604230 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.604207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"8cfa0538841fb2473e270316f4fd1711c697f4452b12b92bb792e7377e7f8ffb"} Jul 22 16:12:27.604324 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.604305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"72531984020a25e3f3a13dd3284c474547bcc6f7543926ab135c67bd945da713"} Jul 22 16:12:27.605610 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.605588 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea6724a16869ee2376effcabe611a70e" containerID="7120e23f7c6e6949433afa1cd83f9ea30a30f87f7045613c8cf5c3e35487210e" exitCode=0 Jul 22 16:12:27.605701 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.605664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" event={"ID":"ea6724a16869ee2376effcabe611a70e","Type":"ContainerDied","Data":"7120e23f7c6e6949433afa1cd83f9ea30a30f87f7045613c8cf5c3e35487210e"} Jul 22 16:12:27.607589 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.607537 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t46jb" event={"ID":"937727a3-12c8-486f-b385-a19d18eef8dd","Type":"ContainerStarted","Data":"5f3843575b58e7d146af82b6243fe052f8e793a5deff11b2a5c42295da4b04b7"} Jul 22 16:12:27.609254 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.609230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wcksj" event={"ID":"1a83e2e3-3411-44ab-bd64-6fefd51c5a3c","Type":"ContainerStarted","Data":"c5ab0c888a9d107f0b46cf6707064995ddbe7c28fb6cebe2393d8369256c2cb4"} Jul 22 16:12:27.610899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.610876 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" event={"ID":"dc2ff436-4fc2-4f0f-882c-e290d93b85c8","Type":"ContainerStarted","Data":"07448b8944b2c7b4fe5ae5398885dd66719c8ce40db1a719c9cf73891257e07c"} Jul 22 16:12:27.613161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.613128 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lhwcp" podStartSLOduration=3.115920537 podStartE2EDuration="22.613117763s" totalImagesPullingTime="19.497197226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:27.612894054 +0000 UTC m=+22.597778058" watchObservedRunningTime="2026-07-22 16:12:27.613117763 +0000 UTC m=+22.598001769" Jul 22 16:12:27.633974 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.633938 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wcksj" podStartSLOduration=3.11648209 podStartE2EDuration="22.633926828s" totalImagesPullingTime="19.517444738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:27.633639212 +0000 UTC m=+22.618523219" watchObservedRunningTime="2026-07-22 16:12:27.633926828 +0000 UTC m=+22.618810834" Jul 22 16:12:27.654177 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.653954 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-782hj" podStartSLOduration=3.15017347 podStartE2EDuration="22.653942396s" totalImagesPullingTime="19.503768926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:27.653210778 +0000 UTC m=+22.638094785" watchObservedRunningTime="2026-07-22 16:12:27.653942396 +0000 UTC m=+22.638826403" Jul 22 16:12:27.677748 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.677465 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4hj59" podStartSLOduration=3.157077894 podStartE2EDuration="22.677450872s" totalImagesPullingTime="19.520372978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:27.676602669 +0000 UTC m=+22.661486677" watchObservedRunningTime="2026-07-22 16:12:27.677450872 +0000 UTC m=+22.662334879" Jul 22 16:12:27.733505 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.733469 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t46jb" podStartSLOduration=2.67089258 podStartE2EDuration="22.733456879s" totalImagesPullingTime="20.062564299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:27.732420263 +0000 UTC m=+22.717304271" watchObservedRunningTime="2026-07-22 16:12:27.733456879 +0000 UTC m=+22.718340884" Jul 22 16:12:27.765154 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.765130 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:27.765585 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.765568 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:27.767662 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.767614 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:27.855444 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.855367 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:27.952865 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:27.952840 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:28.057174 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.057148 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:28.140571 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.140488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:28.339673 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.339649 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 16:12:28.396220 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.396056 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T16:12:28.339670182Z","UUID":"7fd38cb3-38ef-4a22-b5f9-9a1232734853","Handler":null,"Name":"","Endpoint":""} Jul 22 16:12:28.397462 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.397445 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 16:12:28.397581 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.397485 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 16:12:28.485342 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.485307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:28.485486 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:28.485418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:28.614891 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.614855 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" event={"ID":"ea6724a16869ee2376effcabe611a70e","Type":"ContainerStarted","Data":"2467e0746dfd556019eb6a7484018ba5235380d1f93b3b2a40f7be793e57ef37"} Jul 22 16:12:28.617187 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.617150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" event={"ID":"dc2ff436-4fc2-4f0f-882c-e290d93b85c8","Type":"ContainerStarted","Data":"f335762bf8e1a510a3f17f9fd0b90ba4ddf7e2d8a23fb0e0d4858afcaec59880"} Jul 22 16:12:28.617427 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.617400 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:28.617982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.617966 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lhwcp" Jul 22 16:12:28.635246 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.635208 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-1.ec2.internal" podStartSLOduration=22.635192862 podStartE2EDuration="22.635192862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:28.63489151 +0000 UTC m=+23.619775516" watchObservedRunningTime="2026-07-22 16:12:28.635192862 +0000 UTC m=+23.620076879" Jul 22 16:12:28.736062 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:28.735989 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:29.489131 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:29.489067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:29.489385 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:29.489081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:29.489385 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:29.489192 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:29.489385 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:29.489248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:29.623307 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:29.623288 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:12:29.623783 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:29.623761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"bdd88978d91d72cf8b108948b1970dc6adcc59e3abebaaae788244694f86c533"} Jul 22 16:12:30.485753 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:30.485727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:30.486007 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:30.485842 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:30.627676 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:30.627640 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" event={"ID":"dc2ff436-4fc2-4f0f-882c-e290d93b85c8","Type":"ContainerStarted","Data":"d2d5b3fa45ed6d4f668a7bc1eb69fe9660164cd5d56a2ba7d917e823f9cc9ac6"} Jul 22 16:12:31.485677 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:31.485598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:31.485820 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:31.485598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:31.485820 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:31.485722 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:31.485820 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:31.485757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:32.486241 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.486041 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:32.486865 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:32.486274 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:32.632847 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.632815 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2076767-84a2-4ba2-b556-30c266b278b6" containerID="d221e2d3373e37f140a431d3a43e5197689b5d75ecbc3fb34806203a18cd0f52" exitCode=0 Jul 22 16:12:32.632984 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.632876 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerDied","Data":"d221e2d3373e37f140a431d3a43e5197689b5d75ecbc3fb34806203a18cd0f52"} Jul 22 16:12:32.636563 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.636548 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:12:32.636942 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.636915 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"d85a73b245a0e0b609b35d76cea0f684ebbec205dfdcb92172c9a77ed2b8823b"} Jul 22 16:12:32.637225 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.637211 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:32.637287 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.637268 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:32.637341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.637308 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:32.637393 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.637355 2578 scope.go:117] "RemoveContainer" containerID="6b5a9cee357e7e1016f2c938d0105e5a2f12e87a46de2e8d55ee487922f86455" Jul 22 16:12:32.652390 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.652369 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:32.653126 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.653113 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:12:32.662151 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:32.661791 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6qpf" podStartSLOduration=5.663861412 podStartE2EDuration="27.661775229s" totalImagesPullingTime="21.997913817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:30.65414283 +0000 UTC m=+25.639026837" watchObservedRunningTime="2026-07-22 16:12:32.661775229 +0000 UTC m=+27.646659236" Jul 22 16:12:33.488492 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:33.488463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:33.488874 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:33.488586 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:33.488874 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:33.488634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:33.488874 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:33.488718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:33.643400 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:33.643205 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:12:33.643814 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:33.643783 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" event={"ID":"384883b4-4ca8-4839-8607-4f9536c3ebf2","Type":"ContainerStarted","Data":"7331d750bee3ba74ef54e452b0f9e1970d8f3d01dd926606607387328c2759d9"} Jul 22 16:12:33.681749 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:33.681707 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" podStartSLOduration=9.051098482 podStartE2EDuration="28.681692284s" totalImagesPullingTime="19.630593802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:33.68101712 +0000 UTC m=+28.665901150" watchObservedRunningTime="2026-07-22 16:12:33.681692284 +0000 UTC m=+28.666576289" Jul 22 16:12:34.043383 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.043350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:34.117873 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.117844 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:34.251807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.251777 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpxss"] Jul 22 16:12:34.251928 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.251878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:34.251989 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:34.251955 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:34.254538 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.254498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z8drf"] Jul 22 16:12:34.254673 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.254631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:34.254769 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:34.254734 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:34.255139 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.255120 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwwp5"] Jul 22 16:12:34.255206 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.255192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:34.255280 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:34.255265 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:34.650878 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.650843 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2076767-84a2-4ba2-b556-30c266b278b6" containerID="0ce2fed25da2b9d139a59df1e2012bc987f20b5c2dfbba5e751e96334a153c79" exitCode=0 Jul 22 16:12:34.651232 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:34.650933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerDied","Data":"0ce2fed25da2b9d139a59df1e2012bc987f20b5c2dfbba5e751e96334a153c79"} Jul 22 16:12:35.488376 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:35.488305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:35.488583 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:35.488426 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:36.485580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:36.485504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:36.485907 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:36.485504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:36.485907 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:36.485621 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:36.485907 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:36.485686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:36.656581 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:36.656553 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2076767-84a2-4ba2-b556-30c266b278b6" containerID="0dc7c53adb3876f6c8fce95f704fdd2ba5cda369ae17024eaea0544ce2c84288" exitCode=0 Jul 22 16:12:36.656712 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:36.656609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerDied","Data":"0dc7c53adb3876f6c8fce95f704fdd2ba5cda369ae17024eaea0544ce2c84288"} Jul 22 16:12:37.485768 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:37.485738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:37.486248 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:37.485851 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z8drf" podUID="1c68118a-8c93-478a-9780-3852ef7091ab" Jul 22 16:12:37.584643 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:37.584587 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:38.174680 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:38.174597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:38.174680 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:38.174653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:38.174931 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.174759 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:38.174931 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.174819 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:13:10.174804862 +0000 UTC m=+65.159688846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:12:38.174931 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.174759 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:38.174931 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.174903 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret podName:1c68118a-8c93-478a-9780-3852ef7091ab nodeName:}" failed. No retries permitted until 2026-07-22 16:13:10.174884144 +0000 UTC m=+65.159768140 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret") pod "global-pull-secret-syncer-z8drf" (UID: "1c68118a-8c93-478a-9780-3852ef7091ab") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:12:38.275064 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:38.275035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:38.275207 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.275172 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:12:38.275207 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.275190 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:12:38.275207 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.275200 2578 projected.go:196] Error preparing data for projected volume kube-api-access-64r7p for pod openshift-network-diagnostics/network-check-target-gpxss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:38.275358 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.275248 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p podName:9969950d-f2c9-424c-9b21-1e6ffd76789d nodeName:}" failed. No retries permitted until 2026-07-22 16:13:10.27523425 +0000 UTC m=+65.260118250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-64r7p" (UniqueName: "kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p") pod "network-check-target-gpxss" (UID: "9969950d-f2c9-424c-9b21-1e6ffd76789d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:12:38.486194 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:38.486116 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:38.486678 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:38.486116 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:38.486678 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.486245 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwwp5" podUID="06184455-9451-417f-9a46-2009651babfc" Jul 22 16:12:38.486678 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:38.486341 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpxss" podUID="9969950d-f2c9-424c-9b21-1e6ffd76789d" Jul 22 16:12:39.297596 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.297397 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-1.ec2.internal" event="NodeReady" Jul 22 16:12:39.297734 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.297694 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 16:12:39.347447 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.347418 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4mrzj"] Jul 22 16:12:39.368611 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.368586 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m56n4"] Jul 22 16:12:39.368761 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.368719 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.371553 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.371471 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 16:12:39.371685 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.371582 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 16:12:39.371835 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.371502 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bj2bw\"" Jul 22 16:12:39.381664 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.381644 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4mrzj"] Jul 22 16:12:39.381753 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.381741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:39.385234 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.385215 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 16:12:39.385330 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.385215 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hg4lc\"" Jul 22 16:12:39.385762 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.385746 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 16:12:39.386683 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.386658 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 16:12:39.390905 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.390828 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m56n4"] Jul 22 16:12:39.484944 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.484864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2873ca4a-21f4-4440-8f1e-1eac30996495-cert\") pod \"ingress-canary-m56n4\" (UID: \"2873ca4a-21f4-4440-8f1e-1eac30996495\") " pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:39.484944 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.484908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64pfw\" (UniqueName: \"kubernetes.io/projected/4102150a-5b4f-4905-b488-b86f1c7e56fb-kube-api-access-64pfw\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.485131 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.485002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.485131 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.485052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4102150a-5b4f-4905-b488-b86f1c7e56fb-config-volume\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.485131 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.485100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjm5\" (UniqueName: \"kubernetes.io/projected/2873ca4a-21f4-4440-8f1e-1eac30996495-kube-api-access-rwjm5\") pod \"ingress-canary-m56n4\" (UID: \"2873ca4a-21f4-4440-8f1e-1eac30996495\") " pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:39.485273 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.485144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4102150a-5b4f-4905-b488-b86f1c7e56fb-tmp-dir\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.485318 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.485305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:12:39.488118 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.488088 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:12:39.529644 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.529595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:39.586347 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.586319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4102150a-5b4f-4905-b488-b86f1c7e56fb-tmp-dir\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.586473 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.586359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2873ca4a-21f4-4440-8f1e-1eac30996495-cert\") pod \"ingress-canary-m56n4\" (UID: \"2873ca4a-21f4-4440-8f1e-1eac30996495\") " pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:39.586473 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.586386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64pfw\" (UniqueName: \"kubernetes.io/projected/4102150a-5b4f-4905-b488-b86f1c7e56fb-kube-api-access-64pfw\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.586473 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.586445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.586708 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.586476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4102150a-5b4f-4905-b488-b86f1c7e56fb-config-volume\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.586708 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.586504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjm5\" (UniqueName: \"kubernetes.io/projected/2873ca4a-21f4-4440-8f1e-1eac30996495-kube-api-access-rwjm5\") pod \"ingress-canary-m56n4\" (UID: \"2873ca4a-21f4-4440-8f1e-1eac30996495\") " pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:39.586708 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:39.586597 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:12:39.586708 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:39.586658 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls podName:4102150a-5b4f-4905-b488-b86f1c7e56fb nodeName:}" failed. No retries permitted until 2026-07-22 16:12:40.086638383 +0000 UTC m=+35.071522384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls") pod "dns-default-4mrzj" (UID: "4102150a-5b4f-4905-b488-b86f1c7e56fb") : secret "dns-default-metrics-tls" not found Jul 22 16:12:39.586708 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.586686 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4102150a-5b4f-4905-b488-b86f1c7e56fb-tmp-dir\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.587117 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.587099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4102150a-5b4f-4905-b488-b86f1c7e56fb-config-volume\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.591142 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.591121 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2873ca4a-21f4-4440-8f1e-1eac30996495-cert\") pod \"ingress-canary-m56n4\" (UID: \"2873ca4a-21f4-4440-8f1e-1eac30996495\") " pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:39.597044 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.597018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64pfw\" (UniqueName: \"kubernetes.io/projected/4102150a-5b4f-4905-b488-b86f1c7e56fb-kube-api-access-64pfw\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:39.597293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.597276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjm5\" (UniqueName: \"kubernetes.io/projected/2873ca4a-21f4-4440-8f1e-1eac30996495-kube-api-access-rwjm5\") pod \"ingress-canary-m56n4\" (UID: \"2873ca4a-21f4-4440-8f1e-1eac30996495\") " pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:39.690990 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:39.690960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m56n4" Jul 22 16:12:40.090036 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.089995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:40.090218 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:40.090180 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:12:40.090290 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:40.090259 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls podName:4102150a-5b4f-4905-b488-b86f1c7e56fb nodeName:}" failed. No retries permitted until 2026-07-22 16:12:41.090235934 +0000 UTC m=+36.075119925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls") pod "dns-default-4mrzj" (UID: "4102150a-5b4f-4905-b488-b86f1c7e56fb") : secret "dns-default-metrics-tls" not found Jul 22 16:12:40.485580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.485494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:12:40.485580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.485547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:12:40.489737 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.489711 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 16:12:40.489737 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.489729 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 16:12:40.490200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.489767 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wvt7k\"" Jul 22 16:12:40.490200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.489784 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-df68t\"" Jul 22 16:12:40.490200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.489729 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:12:40.770166 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:40.769984 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m56n4"] Jul 22 16:12:40.777357 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:40.777329 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2873ca4a_21f4_4440_8f1e_1eac30996495.slice/crio-be3e2fcb6508da47089fd20ed2cacfc48c471b1e6c727ca94b10baa8cd029b3b WatchSource:0}: Error finding container be3e2fcb6508da47089fd20ed2cacfc48c471b1e6c727ca94b10baa8cd029b3b: Status 404 returned error can't find the container with id be3e2fcb6508da47089fd20ed2cacfc48c471b1e6c727ca94b10baa8cd029b3b Jul 22 16:12:41.096815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:41.096710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:41.096943 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:41.096845 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:12:41.096943 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:41.096904 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls podName:4102150a-5b4f-4905-b488-b86f1c7e56fb nodeName:}" failed. No retries permitted until 2026-07-22 16:12:43.096887135 +0000 UTC m=+38.081771126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls") pod "dns-default-4mrzj" (UID: "4102150a-5b4f-4905-b488-b86f1c7e56fb") : secret "dns-default-metrics-tls" not found Jul 22 16:12:41.669144 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:41.669098 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m56n4" event={"ID":"2873ca4a-21f4-4440-8f1e-1eac30996495","Type":"ContainerStarted","Data":"be3e2fcb6508da47089fd20ed2cacfc48c471b1e6c727ca94b10baa8cd029b3b"} Jul 22 16:12:41.672182 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:41.672153 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2076767-84a2-4ba2-b556-30c266b278b6" containerID="383b7ac2ea2e213b353972c21ad48a4b794c5e7c66e77bde6b959e2c9bfc8d04" exitCode=0 Jul 22 16:12:41.672277 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:41.672204 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerDied","Data":"383b7ac2ea2e213b353972c21ad48a4b794c5e7c66e77bde6b959e2c9bfc8d04"} Jul 22 16:12:42.678163 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:42.678133 2578 generic.go:356] "Generic (PLEG): container finished" podID="a2076767-84a2-4ba2-b556-30c266b278b6" containerID="33c62fffbf46693491a0e1697e91b4a0b07e180755eabf265af99afd47860a82" exitCode=0 Jul 22 16:12:42.678577 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:42.678172 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerDied","Data":"33c62fffbf46693491a0e1697e91b4a0b07e180755eabf265af99afd47860a82"} Jul 22 16:12:43.111326 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.111234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:43.111487 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:43.111388 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:12:43.111487 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:43.111457 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls podName:4102150a-5b4f-4905-b488-b86f1c7e56fb nodeName:}" failed. No retries permitted until 2026-07-22 16:12:47.111439935 +0000 UTC m=+42.096323919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls") pod "dns-default-4mrzj" (UID: "4102150a-5b4f-4905-b488-b86f1c7e56fb") : secret "dns-default-metrics-tls" not found Jul 22 16:12:43.438054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.437972 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:43.508516 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.508476 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:43.681510 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.681471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m56n4" event={"ID":"2873ca4a-21f4-4440-8f1e-1eac30996495","Type":"ContainerStarted","Data":"956f4d423a7aa757899290cd4b0d7d3a681e19369e906acc6ceec05e3d68bce4"} Jul 22 16:12:43.685253 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.685228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rns6" event={"ID":"a2076767-84a2-4ba2-b556-30c266b278b6","Type":"ContainerStarted","Data":"829731a08d5242548856343fe9d6747b1a13172fb5d3a56f69175fb4fc56292d"} Jul 22 16:12:43.704417 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.704331 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m56n4" podStartSLOduration=2.220299931 podStartE2EDuration="4.704313996s" totalImagesPullingTime="2.484014065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:43.704072233 +0000 UTC m=+38.688956238" watchObservedRunningTime="2026-07-22 16:12:43.704313996 +0000 UTC m=+38.689198002" Jul 22 16:12:43.729816 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.729771 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5rns6" podStartSLOduration=8.847750921 podStartE2EDuration="38.72973659s" totalImagesPullingTime="29.881985669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:43.728706539 +0000 UTC m=+38.713590544" watchObservedRunningTime="2026-07-22 16:12:43.72973659 +0000 UTC m=+38.714620577" Jul 22 16:12:43.856970 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.856938 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:43.939492 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:43.939460 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:44.953920 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:44.953892 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:45.048184 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.048152 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:45.647302 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.647274 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:45.719175 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.719150 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t"] Jul 22 16:12:45.743690 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.743668 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t"] Jul 22 16:12:45.743814 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.743765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.746736 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.746713 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:45.748241 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.748221 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 16:12:45.748591 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.748573 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 16:12:45.750960 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.750916 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gktdl\"" Jul 22 16:12:45.751167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.751151 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 16:12:45.751249 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.751214 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 16:12:45.828917 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.828890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.829003 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.828930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.829003 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.828961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2n4\" (UniqueName: \"kubernetes.io/projected/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-kube-api-access-fm2n4\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.929912 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.929848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.929912 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.929888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.930030 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.929919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2n4\" (UniqueName: \"kubernetes.io/projected/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-kube-api-access-fm2n4\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.930030 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:45.929998 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:45.930109 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:45.930068 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls podName:854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b nodeName:}" failed. No retries permitted until 2026-07-22 16:12:46.43004842 +0000 UTC m=+41.414932418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zmt6t" (UID: "854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:45.930614 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.930597 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:45.934871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.934841 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:45.948126 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:45.948099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2n4\" (UniqueName: \"kubernetes.io/projected/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-kube-api-access-fm2n4\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:46.036362 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:46.036329 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:46.433641 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:46.433608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:46.433810 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:46.433759 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:46.433855 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:46.433845 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls podName:854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b nodeName:}" failed. No retries permitted until 2026-07-22 16:12:47.433827786 +0000 UTC m=+42.418711769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zmt6t" (UID: "854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:47.136871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:47.136840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:47.137238 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:47.136990 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:12:47.137238 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:47.137056 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls podName:4102150a-5b4f-4905-b488-b86f1c7e56fb nodeName:}" failed. No retries permitted until 2026-07-22 16:12:55.13704065 +0000 UTC m=+50.121924633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls") pod "dns-default-4mrzj" (UID: "4102150a-5b4f-4905-b488-b86f1c7e56fb") : secret "dns-default-metrics-tls" not found Jul 22 16:12:47.439230 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:47.439161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:47.439364 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:47.439303 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:47.439734 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:47.439712 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls podName:854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b nodeName:}" failed. No retries permitted until 2026-07-22 16:12:49.439675678 +0000 UTC m=+44.424559665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zmt6t" (UID: "854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:49.452175 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:49.452139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:49.452603 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:49.452263 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:49.452603 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:49.452330 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls podName:854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b nodeName:}" failed. No retries permitted until 2026-07-22 16:12:53.452315899 +0000 UTC m=+48.437199883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zmt6t" (UID: "854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:49.996395 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:49.996363 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6qnpq"] Jul 22 16:12:50.037480 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.037455 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qnpq"] Jul 22 16:12:50.037611 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.037567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.040600 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.040574 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 16:12:50.040721 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.040622 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 16:12:50.041890 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.041872 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 16:12:50.041980 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.041905 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88nqz\"" Jul 22 16:12:50.041980 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.041952 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 16:12:50.088312 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.088276 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:50.155429 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.155407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-data-volume\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.155513 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.155434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.155513 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.155469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.155602 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.155586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-crio-socket\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.155636 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.155613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdcqg\" (UniqueName: \"kubernetes.io/projected/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-api-access-zdcqg\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.181517 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.181490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:50.256590 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.256541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-data-volume\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.256590 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.256568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.256707 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.256601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.256707 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.256659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-crio-socket\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.256707 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.256689 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:12:50.256823 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.256749 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:12:50.256823 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.256765 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:50.756746867 +0000 UTC m=+45.741630855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : secret "insights-runtime-extractor-tls" not found Jul 22 16:12:50.256823 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.256688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdcqg\" (UniqueName: \"kubernetes.io/projected/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-api-access-zdcqg\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.256823 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.256803 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:50.756786773 +0000 UTC m=+45.741670759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : configmap "kube-rbac-proxy" not found Jul 22 16:12:50.256970 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.256862 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-crio-socket\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.256970 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.256928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-data-volume\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.268980 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.268955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdcqg\" (UniqueName: \"kubernetes.io/projected/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-api-access-zdcqg\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.760757 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.760724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.761134 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:50.760770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:50.761134 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.760866 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:12:50.761134 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.760934 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:51.760917981 +0000 UTC m=+46.745801965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : secret "insights-runtime-extractor-tls" not found Jul 22 16:12:50.761134 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.760873 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:12:50.761134 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:50.760991 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:51.760980625 +0000 UTC m=+46.745864612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : configmap "kube-rbac-proxy" not found Jul 22 16:12:51.584289 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.584259 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x"] Jul 22 16:12:51.600731 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.600702 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4"] Jul 22 16:12:51.600849 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.600745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" Jul 22 16:12:51.603840 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.603823 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:12:51.604823 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.604806 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fjswr\"" Jul 22 16:12:51.604895 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.604870 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 16:12:51.622771 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.622751 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x"] Jul 22 16:12:51.622871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.622786 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4"] Jul 22 16:12:51.622871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.622838 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:51.626370 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.626350 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 16:12:51.626370 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.626360 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:12:51.626509 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.626408 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 16:12:51.626671 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.626656 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gqcqp\"" Jul 22 16:12:51.768232 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.768205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtflb\" (UniqueName: \"kubernetes.io/projected/630e27b6-d0f0-4d37-a45b-e0f43872e6c1-kube-api-access-rtflb\") pod \"volume-data-source-validator-78cf4c6f6-7lx9x\" (UID: \"630e27b6-d0f0-4d37-a45b-e0f43872e6c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.768240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.768288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpl9q\" (UniqueName: \"kubernetes.io/projected/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-kube-api-access-lpl9q\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.768335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.768393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:51.768437 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:51.768459 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:51.768497 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:53.768472316 +0000 UTC m=+48.753356301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : secret "insights-runtime-extractor-tls" not found Jul 22 16:12:51.768545 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:51.768513 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:53.768504288 +0000 UTC m=+48.753388277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : configmap "kube-rbac-proxy" not found Jul 22 16:12:51.782493 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.782473 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hj59_0a3df486-be43-4d53-ac36-3e6ad9cfcb80/dns-node-resolver/0.log" Jul 22 16:12:51.869055 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.868999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtflb\" (UniqueName: \"kubernetes.io/projected/630e27b6-d0f0-4d37-a45b-e0f43872e6c1-kube-api-access-rtflb\") pod \"volume-data-source-validator-78cf4c6f6-7lx9x\" (UID: \"630e27b6-d0f0-4d37-a45b-e0f43872e6c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" Jul 22 16:12:51.869055 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.869031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:51.869161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.869064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpl9q\" (UniqueName: \"kubernetes.io/projected/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-kube-api-access-lpl9q\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:51.869161 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:51.869138 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:12:51.869221 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:51.869209 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls podName:c60d99d0-e447-473f-ae4f-cc4d378dc5b5 nodeName:}" failed. No retries permitted until 2026-07-22 16:12:52.369192895 +0000 UTC m=+47.354076887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hrhm4" (UID: "c60d99d0-e447-473f-ae4f-cc4d378dc5b5") : secret "samples-operator-tls" not found Jul 22 16:12:51.881999 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.881973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpl9q\" (UniqueName: \"kubernetes.io/projected/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-kube-api-access-lpl9q\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:51.882203 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.882183 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtflb\" (UniqueName: \"kubernetes.io/projected/630e27b6-d0f0-4d37-a45b-e0f43872e6c1-kube-api-access-rtflb\") pod \"volume-data-source-validator-78cf4c6f6-7lx9x\" (UID: \"630e27b6-d0f0-4d37-a45b-e0f43872e6c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" Jul 22 16:12:51.911212 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:51.911190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" Jul 22 16:12:52.069775 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:52.069721 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x"] Jul 22 16:12:52.073204 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:52.073180 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630e27b6_d0f0_4d37_a45b_e0f43872e6c1.slice/crio-8217fa927132f7bf4f9ab26a17d767530da2160e61f7b37d3324cafb8e53ee1a WatchSource:0}: Error finding container 8217fa927132f7bf4f9ab26a17d767530da2160e61f7b37d3324cafb8e53ee1a: Status 404 returned error can't find the container with id 8217fa927132f7bf4f9ab26a17d767530da2160e61f7b37d3324cafb8e53ee1a Jul 22 16:12:52.372213 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:52.372184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:52.372335 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:52.372318 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:12:52.372397 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:52.372389 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls podName:c60d99d0-e447-473f-ae4f-cc4d378dc5b5 nodeName:}" failed. No retries permitted until 2026-07-22 16:12:53.372368917 +0000 UTC m=+48.357252911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hrhm4" (UID: "c60d99d0-e447-473f-ae4f-cc4d378dc5b5") : secret "samples-operator-tls" not found Jul 22 16:12:52.705248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:52.705181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" event={"ID":"630e27b6-d0f0-4d37-a45b-e0f43872e6c1","Type":"ContainerStarted","Data":"8217fa927132f7bf4f9ab26a17d767530da2160e61f7b37d3324cafb8e53ee1a"} Jul 22 16:12:52.985670 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:52.985594 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wcksj_1a83e2e3-3411-44ab-bd64-6fefd51c5a3c/node-ca/0.log" Jul 22 16:12:53.380228 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.380199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:53.380379 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.380313 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:12:53.380379 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.380364 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls podName:c60d99d0-e447-473f-ae4f-cc4d378dc5b5 nodeName:}" failed. No retries permitted until 2026-07-22 16:12:55.380351734 +0000 UTC m=+50.365235718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hrhm4" (UID: "c60d99d0-e447-473f-ae4f-cc4d378dc5b5") : secret "samples-operator-tls" not found Jul 22 16:12:53.481391 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.481359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:12:53.481544 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.481503 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:53.481602 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.481590 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls podName:854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b nodeName:}" failed. No retries permitted until 2026-07-22 16:13:01.481574769 +0000 UTC m=+56.466458756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zmt6t" (UID: "854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:12:53.573966 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.573935 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-smbvj"] Jul 22 16:12:53.586548 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.586509 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m56n4_2873ca4a-21f4-4440-8f1e-1eac30996495/serve-healthcheck-canary/0.log" Jul 22 16:12:53.594854 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.594834 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-smbvj"] Jul 22 16:12:53.594971 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.594930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.597763 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.597742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 16:12:53.597878 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.597794 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 16:12:53.597878 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.597797 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 16:12:53.598919 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.598900 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:12:53.599018 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.598979 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dstqb\"" Jul 22 16:12:53.603362 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.603341 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 16:12:53.784027 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.784005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7c8\" (UniqueName: \"kubernetes.io/projected/a285f20a-93c6-4fd8-9780-49d61469ef5e-kube-api-access-9r7c8\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.784128 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.784078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:53.784128 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.784108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a285f20a-93c6-4fd8-9780-49d61469ef5e-config\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.784128 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.784126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a285f20a-93c6-4fd8-9780-49d61469ef5e-serving-cert\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.784295 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.784177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:53.784295 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.784206 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:12:53.784295 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.784236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a285f20a-93c6-4fd8-9780-49d61469ef5e-trusted-ca\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.784295 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.784258 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:57.784240216 +0000 UTC m=+52.769124208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : secret "insights-runtime-extractor-tls" not found Jul 22 16:12:53.784295 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.784280 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:12:53.784596 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:53.784333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:12:57.784317119 +0000 UTC m=+52.769201103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : configmap "kube-rbac-proxy" not found Jul 22 16:12:53.885451 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.885430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a285f20a-93c6-4fd8-9780-49d61469ef5e-config\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.885572 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.885457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a285f20a-93c6-4fd8-9780-49d61469ef5e-serving-cert\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.885572 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.885556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a285f20a-93c6-4fd8-9780-49d61469ef5e-trusted-ca\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.885688 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.885613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7c8\" (UniqueName: \"kubernetes.io/projected/a285f20a-93c6-4fd8-9780-49d61469ef5e-kube-api-access-9r7c8\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.886056 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.886034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a285f20a-93c6-4fd8-9780-49d61469ef5e-config\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.886283 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.886265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a285f20a-93c6-4fd8-9780-49d61469ef5e-trusted-ca\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.887561 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.887542 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a285f20a-93c6-4fd8-9780-49d61469ef5e-serving-cert\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.894848 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.894828 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7c8\" (UniqueName: \"kubernetes.io/projected/a285f20a-93c6-4fd8-9780-49d61469ef5e-kube-api-access-9r7c8\") pod \"console-operator-565b7b569d-smbvj\" (UID: \"a285f20a-93c6-4fd8-9780-49d61469ef5e\") " pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:53.904680 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:53.904662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:12:54.040453 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.040423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-smbvj"] Jul 22 16:12:54.043411 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:12:54.043384 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda285f20a_93c6_4fd8_9780_49d61469ef5e.slice/crio-c3066e7a4ec6396eb0759fa5e55c072388a1cc4ee57f7ec0f3cfb74d735cf5df WatchSource:0}: Error finding container c3066e7a4ec6396eb0759fa5e55c072388a1cc4ee57f7ec0f3cfb74d735cf5df: Status 404 returned error can't find the container with id c3066e7a4ec6396eb0759fa5e55c072388a1cc4ee57f7ec0f3cfb74d735cf5df Jul 22 16:12:54.415277 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.415208 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58bd944b45-r42jf"] Jul 22 16:12:54.424255 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.424236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.427055 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.427036 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 16:12:54.427905 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.427887 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lcgb9\"" Jul 22 16:12:54.428160 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.428141 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 16:12:54.428224 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.428169 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 16:12:54.435390 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.435363 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58bd944b45-r42jf"] Jul 22 16:12:54.436224 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.436207 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 16:12:54.590427 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-registry-certificates\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.590552 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-bound-sa-token\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.590552 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtgvp\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-kube-api-access-jtgvp\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.590637 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-image-registry-private-configuration\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.590637 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.590637 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-trusted-ca\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.590747 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2946fec9-cc76-45ff-ba02-077511935015-ca-trust-extracted\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.590747 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.590693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-installation-pull-secrets\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691193 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691128 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2946fec9-cc76-45ff-ba02-077511935015-ca-trust-extracted\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691193 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-installation-pull-secrets\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691331 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-registry-certificates\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691331 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-bound-sa-token\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691331 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtgvp\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-kube-api-access-jtgvp\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691331 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-image-registry-private-configuration\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691473 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691473 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-trusted-ca\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691588 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.691568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2946fec9-cc76-45ff-ba02-077511935015-ca-trust-extracted\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.691717 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:54.691696 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:12:54.691789 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:54.691718 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bd944b45-r42jf: secret "image-registry-tls" not found Jul 22 16:12:54.691789 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:54.691777 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls podName:2946fec9-cc76-45ff-ba02-077511935015 nodeName:}" failed. No retries permitted until 2026-07-22 16:12:55.191758255 +0000 UTC m=+50.176642255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls") pod "image-registry-58bd944b45-r42jf" (UID: "2946fec9-cc76-45ff-ba02-077511935015") : secret "image-registry-tls" not found Jul 22 16:12:54.692354 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.692324 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-registry-certificates\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.692444 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.692362 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-trusted-ca\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.693908 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.693889 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-installation-pull-secrets\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.694313 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.694293 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-image-registry-private-configuration\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.705162 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.705144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-bound-sa-token\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.705248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.705219 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtgvp\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-kube-api-access-jtgvp\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:54.711740 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.711711 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" event={"ID":"630e27b6-d0f0-4d37-a45b-e0f43872e6c1","Type":"ContainerStarted","Data":"2bb02bdb44d37917f6ce4ccfc6e2331af7116a4d5d0e3935ee43a055be627590"} Jul 22 16:12:54.713012 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.712993 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" event={"ID":"a285f20a-93c6-4fd8-9780-49d61469ef5e","Type":"ContainerStarted","Data":"c3066e7a4ec6396eb0759fa5e55c072388a1cc4ee57f7ec0f3cfb74d735cf5df"} Jul 22 16:12:54.732209 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.732161 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-7lx9x" podStartSLOduration=2.062416144 podStartE2EDuration="3.732148376s" totalImagesPullingTime="1.669732232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:12:54.731250126 +0000 UTC m=+49.716134134" watchObservedRunningTime="2026-07-22 16:12:54.732148376 +0000 UTC m=+49.717032381" Jul 22 16:12:54.848370 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.848334 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:54.957539 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:54.957442 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:12:55.195551 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:55.195504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:55.195551 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:55.195557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:12:55.195983 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:55.195636 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 16:12:55.195983 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:55.195652 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:12:55.195983 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:55.195671 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bd944b45-r42jf: secret "image-registry-tls" not found Jul 22 16:12:55.195983 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:55.195686 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls podName:4102150a-5b4f-4905-b488-b86f1c7e56fb nodeName:}" failed. No retries permitted until 2026-07-22 16:13:11.195673755 +0000 UTC m=+66.180557742 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls") pod "dns-default-4mrzj" (UID: "4102150a-5b4f-4905-b488-b86f1c7e56fb") : secret "dns-default-metrics-tls" not found Jul 22 16:12:55.195983 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:55.195726 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls podName:2946fec9-cc76-45ff-ba02-077511935015 nodeName:}" failed. No retries permitted until 2026-07-22 16:12:56.195708492 +0000 UTC m=+51.180592488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls") pod "image-registry-58bd944b45-r42jf" (UID: "2946fec9-cc76-45ff-ba02-077511935015") : secret "image-registry-tls" not found Jul 22 16:12:55.397792 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:55.397767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:55.397959 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:55.397905 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:12:55.397959 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:55.397956 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls podName:c60d99d0-e447-473f-ae4f-cc4d378dc5b5 nodeName:}" failed. No retries permitted until 2026-07-22 16:12:59.397942943 +0000 UTC m=+54.382826927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hrhm4" (UID: "c60d99d0-e447-473f-ae4f-cc4d378dc5b5") : secret "samples-operator-tls" not found Jul 22 16:12:56.205591 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:56.205554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:56.206037 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:56.205737 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:12:56.206037 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:56.205761 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bd944b45-r42jf: secret "image-registry-tls" not found Jul 22 16:12:56.206037 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:56.205843 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls podName:2946fec9-cc76-45ff-ba02-077511935015 nodeName:}" failed. No retries permitted until 2026-07-22 16:12:58.205821152 +0000 UTC m=+53.190705158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls") pod "image-registry-58bd944b45-r42jf" (UID: "2946fec9-cc76-45ff-ba02-077511935015") : secret "image-registry-tls" not found Jul 22 16:12:57.722833 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:57.722800 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/0.log" Jul 22 16:12:57.723295 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:57.722844 2578 generic.go:356] "Generic (PLEG): container finished" podID="a285f20a-93c6-4fd8-9780-49d61469ef5e" containerID="2d0a50cc13f8228adede55691cb27df2ed91847ad1104539746525bc2c5a0cd2" exitCode=255 Jul 22 16:12:57.723295 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:57.722903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" event={"ID":"a285f20a-93c6-4fd8-9780-49d61469ef5e","Type":"ContainerDied","Data":"2d0a50cc13f8228adede55691cb27df2ed91847ad1104539746525bc2c5a0cd2"} Jul 22 16:12:57.723295 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:57.723161 2578 scope.go:117] "RemoveContainer" containerID="2d0a50cc13f8228adede55691cb27df2ed91847ad1104539746525bc2c5a0cd2" Jul 22 16:12:57.818074 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:57.818048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:57.818181 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:57.818167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:12:57.818249 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:57.818187 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:12:57.818249 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:57.818238 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 16:12:57.818249 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:57.818246 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:13:05.81822826 +0000 UTC m=+60.803112260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : secret "insights-runtime-extractor-tls" not found Jul 22 16:12:57.818392 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:57.818286 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:13:05.81827026 +0000 UTC m=+60.803154244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : configmap "kube-rbac-proxy" not found Jul 22 16:12:58.220445 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:58.220414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:12:58.220673 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:58.220568 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:12:58.220673 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:58.220586 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bd944b45-r42jf: secret "image-registry-tls" not found Jul 22 16:12:58.220673 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:58.220639 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls podName:2946fec9-cc76-45ff-ba02-077511935015 nodeName:}" failed. No retries permitted until 2026-07-22 16:13:02.220623642 +0000 UTC m=+57.205507627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls") pod "image-registry-58bd944b45-r42jf" (UID: "2946fec9-cc76-45ff-ba02-077511935015") : secret "image-registry-tls" not found Jul 22 16:12:58.726851 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:58.726826 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/1.log" Jul 22 16:12:58.727237 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:58.727223 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/0.log" Jul 22 16:12:58.727290 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:58.727256 2578 generic.go:356] "Generic (PLEG): container finished" podID="a285f20a-93c6-4fd8-9780-49d61469ef5e" containerID="3a60038d3874e1466aaca526d440214bae7ab2879708194b38d50f548863527d" exitCode=255 Jul 22 16:12:58.727326 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:58.727296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" event={"ID":"a285f20a-93c6-4fd8-9780-49d61469ef5e","Type":"ContainerDied","Data":"3a60038d3874e1466aaca526d440214bae7ab2879708194b38d50f548863527d"} Jul 22 16:12:58.727362 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:58.727352 2578 scope.go:117] "RemoveContainer" containerID="2d0a50cc13f8228adede55691cb27df2ed91847ad1104539746525bc2c5a0cd2" Jul 22 16:12:58.727559 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:58.727513 2578 scope.go:117] "RemoveContainer" containerID="3a60038d3874e1466aaca526d440214bae7ab2879708194b38d50f548863527d" Jul 22 16:12:58.727724 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:58.727702 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-smbvj_openshift-console-operator(a285f20a-93c6-4fd8-9780-49d61469ef5e)\"" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" podUID="a285f20a-93c6-4fd8-9780-49d61469ef5e" Jul 22 16:12:59.429739 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.429706 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:12:59.429895 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:59.429835 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 16:12:59.429895 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:59.429889 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls podName:c60d99d0-e447-473f-ae4f-cc4d378dc5b5 nodeName:}" failed. No retries permitted until 2026-07-22 16:13:07.429875015 +0000 UTC m=+62.414759003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-hrhm4" (UID: "c60d99d0-e447-473f-ae4f-cc4d378dc5b5") : secret "samples-operator-tls" not found Jul 22 16:12:59.544638 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.544609 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9"] Jul 22 16:12:59.575110 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.575082 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9"] Jul 22 16:12:59.575242 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.575180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" Jul 22 16:12:59.578008 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.577989 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-djwj4\"" Jul 22 16:12:59.630844 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.630823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9kl\" (UniqueName: \"kubernetes.io/projected/3ad34aa2-eaab-4523-85d5-73ed97adac32-kube-api-access-bq9kl\") pod \"network-check-source-7bc5b6bf85-phrs9\" (UID: \"3ad34aa2-eaab-4523-85d5-73ed97adac32\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" Jul 22 16:12:59.731067 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.731012 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/1.log" Jul 22 16:12:59.731425 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.731238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9kl\" (UniqueName: \"kubernetes.io/projected/3ad34aa2-eaab-4523-85d5-73ed97adac32-kube-api-access-bq9kl\") pod \"network-check-source-7bc5b6bf85-phrs9\" (UID: \"3ad34aa2-eaab-4523-85d5-73ed97adac32\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" Jul 22 16:12:59.731425 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.731370 2578 scope.go:117] "RemoveContainer" containerID="3a60038d3874e1466aaca526d440214bae7ab2879708194b38d50f548863527d" Jul 22 16:12:59.731589 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:12:59.731566 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-smbvj_openshift-console-operator(a285f20a-93c6-4fd8-9780-49d61469ef5e)\"" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" podUID="a285f20a-93c6-4fd8-9780-49d61469ef5e" Jul 22 16:12:59.744718 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.744701 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9kl\" (UniqueName: \"kubernetes.io/projected/3ad34aa2-eaab-4523-85d5-73ed97adac32-kube-api-access-bq9kl\") pod \"network-check-source-7bc5b6bf85-phrs9\" (UID: \"3ad34aa2-eaab-4523-85d5-73ed97adac32\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" Jul 22 16:12:59.883695 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:12:59.883671 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" Jul 22 16:13:00.023029 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.022994 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9"] Jul 22 16:13:00.025453 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:00.025422 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad34aa2_eaab_4523_85d5_73ed97adac32.slice/crio-41ebd9252b4ea8cfec12108aeb9540dac9871c37ffa1d1e04d79c5a08c42adab WatchSource:0}: Error finding container 41ebd9252b4ea8cfec12108aeb9540dac9871c37ffa1d1e04d79c5a08c42adab: Status 404 returned error can't find the container with id 41ebd9252b4ea8cfec12108aeb9540dac9871c37ffa1d1e04d79c5a08c42adab Jul 22 16:13:00.197070 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.197034 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:00.293283 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.293212 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:00.393516 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.393488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:00.489901 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.489868 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:00.735390 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.735306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" event={"ID":"3ad34aa2-eaab-4523-85d5-73ed97adac32","Type":"ContainerStarted","Data":"41ebd9252b4ea8cfec12108aeb9540dac9871c37ffa1d1e04d79c5a08c42adab"} Jul 22 16:13:00.856997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.856963 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:00.932031 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:00.931991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:01.144788 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:01.144757 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:01.237891 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:01.237861 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:01.548006 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:01.547971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:13:01.548169 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:01.548135 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 16:13:01.548224 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:01.548197 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls podName:854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b nodeName:}" failed. No retries permitted until 2026-07-22 16:13:17.548180006 +0000 UTC m=+72.533063990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zmt6t" (UID: "854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b") : secret "cluster-monitoring-operator-tls" not found Jul 22 16:13:02.254452 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:02.254412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:02.254890 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:02.254594 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 16:13:02.254890 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:02.254618 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bd944b45-r42jf: secret "image-registry-tls" not found Jul 22 16:13:02.254890 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:02.254674 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls podName:2946fec9-cc76-45ff-ba02-077511935015 nodeName:}" failed. No retries permitted until 2026-07-22 16:13:10.254657818 +0000 UTC m=+65.239541818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls") pod "image-registry-58bd944b45-r42jf" (UID: "2946fec9-cc76-45ff-ba02-077511935015") : secret "image-registry-tls" not found Jul 22 16:13:02.721892 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:02.721856 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:02.741992 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:02.741962 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" event={"ID":"3ad34aa2-eaab-4523-85d5-73ed97adac32","Type":"ContainerStarted","Data":"f4ae0373dc372dac1ba01c50474b109cdae1101479fc67fb942511077a48b565"} Jul 22 16:13:02.768472 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:02.768426 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-phrs9" podStartSLOduration=1.25874671 podStartE2EDuration="3.768409202s" totalImagesPullingTime="2.509662492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:02.767600968 +0000 UTC m=+57.752484984" watchObservedRunningTime="2026-07-22 16:13:02.768409202 +0000 UTC m=+57.753293211" Jul 22 16:13:02.820456 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:02.820368 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:02.917607 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:02.917576 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:03.026814 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:03.026782 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:03.289127 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:03.289092 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:03.905480 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:03.905441 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:13:03.905663 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:03.905509 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:13:03.905897 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:03.905883 2578 scope.go:117] "RemoveContainer" containerID="3a60038d3874e1466aaca526d440214bae7ab2879708194b38d50f548863527d" Jul 22 16:13:03.906086 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:03.906061 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-smbvj_openshift-console-operator(a285f20a-93c6-4fd8-9780-49d61469ef5e)\"" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" podUID="a285f20a-93c6-4fd8-9780-49d61469ef5e" Jul 22 16:13:03.949869 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:03.949833 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:04.079951 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:04.079919 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:04.199424 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:04.199334 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:04.660762 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:04.660737 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87d4v" Jul 22 16:13:04.872086 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:04.872047 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:05.107153 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.107121 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:05.206890 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.206853 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:05.473424 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.473337 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:05.655240 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.655213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:05.883005 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.882972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:13:05.883457 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.883038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:13:05.883457 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:05.883164 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 16:13:05.883457 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:05.883241 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls podName:d2ab600c-f104-4ad5-a749-f0e5ba6b540a nodeName:}" failed. No retries permitted until 2026-07-22 16:13:21.8832199 +0000 UTC m=+76.868103900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qnpq" (UID: "d2ab600c-f104-4ad5-a749-f0e5ba6b540a") : secret "insights-runtime-extractor-tls" not found Jul 22 16:13:05.883676 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.883616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:13:05.910469 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:05.910439 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:06.133420 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:06.133351 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:06.525957 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:06.525923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:07.126620 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.126580 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:07.496856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.496784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:13:07.499010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.498990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c60d99d0-e447-473f-ae4f-cc4d378dc5b5-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-hrhm4\" (UID: \"c60d99d0-e447-473f-ae4f-cc4d378dc5b5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:13:07.501362 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.501325 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:07.534045 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.534022 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gqcqp\"" Jul 22 16:13:07.541582 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.541556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" Jul 22 16:13:07.606420 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.606369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:07.680774 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.680608 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4"] Jul 22 16:13:07.753387 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.753327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" event={"ID":"c60d99d0-e447-473f-ae4f-cc4d378dc5b5","Type":"ContainerStarted","Data":"361b489dd921d5a113a7b914d7a14baa175ef5f330390f416394488b00eb9469"} Jul 22 16:13:07.944832 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:07.944794 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:08.334205 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:08.334170 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:08.638589 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:08.638507 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:08.738228 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:08.738187 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:08.989362 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:08.989281 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:09.152110 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:09.152081 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:09.548162 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:09.548122 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:09.760968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:09.760943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" event={"ID":"c60d99d0-e447-473f-ae4f-cc4d378dc5b5","Type":"ContainerStarted","Data":"5e68bf42afe22b3a115585c0d0d0158a83c6437e15d4ac8aa495e7de660b7a2f"} Jul 22 16:13:09.958763 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:09.958681 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:10.189319 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.189276 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:10.218968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.218896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:13:10.219098 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.218978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:13:10.221671 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.221654 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:13:10.221729 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.221693 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:13:10.229908 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:10.229889 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 16:13:10.229962 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:10.229942 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs podName:06184455-9451-417f-9a46-2009651babfc nodeName:}" failed. No retries permitted until 2026-07-22 16:14:14.229926874 +0000 UTC m=+129.214810858 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs") pod "network-metrics-daemon-mwwp5" (UID: "06184455-9451-417f-9a46-2009651babfc") : secret "metrics-daemon-secret" not found Jul 22 16:13:10.231904 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.231885 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c68118a-8c93-478a-9780-3852ef7091ab-original-pull-secret\") pod \"global-pull-secret-syncer-z8drf\" (UID: \"1c68118a-8c93-478a-9780-3852ef7091ab\") " pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:13:10.319584 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.319549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:10.319732 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.319612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:13:10.321913 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.321884 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"image-registry-58bd944b45-r42jf\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:10.322038 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.321953 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r7p\" (UniqueName: \"kubernetes.io/projected/9969950d-f2c9-424c-9b21-1e6ffd76789d-kube-api-access-64r7p\") pod \"network-check-target-gpxss\" (UID: \"9969950d-f2c9-424c-9b21-1e6ffd76789d\") " pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:13:10.341901 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.341878 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lcgb9\"" Jul 22 16:13:10.348362 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.348330 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:10.395716 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.395688 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z8drf" Jul 22 16:13:10.496884 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.496853 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58bd944b45-r42jf"] Jul 22 16:13:10.501199 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:10.501168 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2946fec9_cc76_45ff_ba02_077511935015.slice/crio-f1a3ef3baf0161e1a9def91194f3211426e347fdbf4acbde916acf764033df49 WatchSource:0}: Error finding container f1a3ef3baf0161e1a9def91194f3211426e347fdbf4acbde916acf764033df49: Status 404 returned error can't find the container with id f1a3ef3baf0161e1a9def91194f3211426e347fdbf4acbde916acf764033df49 Jul 22 16:13:10.504372 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.504355 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wvt7k\"" Jul 22 16:13:10.512197 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.512179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:13:10.523365 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.523307 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z8drf"] Jul 22 16:13:10.530500 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:10.530479 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c68118a_8c93_478a_9780_3852ef7091ab.slice/crio-751568eef3352bb6f0a77d8458d0ce1ec5aa89d57caade5d2627cda634c74483 WatchSource:0}: Error finding container 751568eef3352bb6f0a77d8458d0ce1ec5aa89d57caade5d2627cda634c74483: Status 404 returned error can't find the container with id 751568eef3352bb6f0a77d8458d0ce1ec5aa89d57caade5d2627cda634c74483 Jul 22 16:13:10.635776 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.635748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpxss"] Jul 22 16:13:10.638835 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:10.638813 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9969950d_f2c9_424c_9b21_1e6ffd76789d.slice/crio-7517c487251af9b0dfced45d63110b9d2ec0aafd88018575bc1a487f7d17fbae WatchSource:0}: Error finding container 7517c487251af9b0dfced45d63110b9d2ec0aafd88018575bc1a487f7d17fbae: Status 404 returned error can't find the container with id 7517c487251af9b0dfced45d63110b9d2ec0aafd88018575bc1a487f7d17fbae Jul 22 16:13:10.747249 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.747163 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:10.765885 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.765857 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z8drf" event={"ID":"1c68118a-8c93-478a-9780-3852ef7091ab","Type":"ContainerStarted","Data":"751568eef3352bb6f0a77d8458d0ce1ec5aa89d57caade5d2627cda634c74483"} Jul 22 16:13:10.767701 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.767675 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" event={"ID":"c60d99d0-e447-473f-ae4f-cc4d378dc5b5","Type":"ContainerStarted","Data":"8b5948e7474b8fa3df5e5ed866c5b72fee7e17578a7a6df12baff2d7dd94e021"} Jul 22 16:13:10.769231 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.769212 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" event={"ID":"2946fec9-cc76-45ff-ba02-077511935015","Type":"ContainerStarted","Data":"3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2"} Jul 22 16:13:10.769327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.769246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" event={"ID":"2946fec9-cc76-45ff-ba02-077511935015","Type":"ContainerStarted","Data":"f1a3ef3baf0161e1a9def91194f3211426e347fdbf4acbde916acf764033df49"} Jul 22 16:13:10.769327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.769286 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:10.770578 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.770554 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpxss" event={"ID":"9969950d-f2c9-424c-9b21-1e6ffd76789d","Type":"ContainerStarted","Data":"88da005182380e23abefd8b31a8d3512c30c7600cac0d5d6dc011c1cf6759e96"} Jul 22 16:13:10.770659 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.770588 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpxss" event={"ID":"9969950d-f2c9-424c-9b21-1e6ffd76789d","Type":"ContainerStarted","Data":"7517c487251af9b0dfced45d63110b9d2ec0aafd88018575bc1a487f7d17fbae"} Jul 22 16:13:10.770708 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.770670 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:13:10.791595 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.791555 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-hrhm4" podStartSLOduration=17.820627649 podStartE2EDuration="19.791540346s" totalImagesPullingTime="1.970912697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:10.79015348 +0000 UTC m=+65.775037485" watchObservedRunningTime="2026-07-22 16:13:10.791540346 +0000 UTC m=+65.776424351" Jul 22 16:13:10.817876 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.817839 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" podStartSLOduration=16.817825779 podStartE2EDuration="16.817825779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:10.817254374 +0000 UTC m=+65.802138381" watchObservedRunningTime="2026-07-22 16:13:10.817825779 +0000 UTC m=+65.802709785" Jul 22 16:13:10.839453 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:10.839419 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gpxss" podStartSLOduration=65.839408015 podStartE2EDuration="1m5.839408015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:10.838642941 +0000 UTC m=+65.823526957" watchObservedRunningTime="2026-07-22 16:13:10.839408015 +0000 UTC m=+65.824292022" Jul 22 16:13:11.173746 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.173709 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:11.227873 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.227832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:13:11.230430 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.230407 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4102150a-5b4f-4905-b488-b86f1c7e56fb-metrics-tls\") pod \"dns-default-4mrzj\" (UID: \"4102150a-5b4f-4905-b488-b86f1c7e56fb\") " pod="openshift-dns/dns-default-4mrzj" Jul 22 16:13:11.320790 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.320752 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:11.482363 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.482286 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bj2bw\"" Jul 22 16:13:11.490074 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.490044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4mrzj" Jul 22 16:13:11.518654 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.518622 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:11.636507 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.636478 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4mrzj"] Jul 22 16:13:11.639781 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:11.639738 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4102150a_5b4f_4905_b488_b86f1c7e56fb.slice/crio-af7eef71f692f2ebbe57d689ede03acac01846ce9ec2e3a13486879ea53fb0d2 WatchSource:0}: Error finding container af7eef71f692f2ebbe57d689ede03acac01846ce9ec2e3a13486879ea53fb0d2: Status 404 returned error can't find the container with id af7eef71f692f2ebbe57d689ede03acac01846ce9ec2e3a13486879ea53fb0d2 Jul 22 16:13:11.733081 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.733006 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:11.775066 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.775033 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4mrzj" event={"ID":"4102150a-5b4f-4905-b488-b86f1c7e56fb","Type":"ContainerStarted","Data":"af7eef71f692f2ebbe57d689ede03acac01846ce9ec2e3a13486879ea53fb0d2"} Jul 22 16:13:11.913145 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:11.913106 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:12.912990 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:12.912957 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:13.329487 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:13.329449 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:13.510054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:13.510022 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:13.734141 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:13.734040 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:13.935415 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:13.935310 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:14.921677 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:14.921644 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:15.118340 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.118301 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:15.298815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.298790 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:15.395501 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.395476 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:15.533011 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.532952 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:15.793674 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.793590 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z8drf" event={"ID":"1c68118a-8c93-478a-9780-3852ef7091ab","Type":"ContainerStarted","Data":"f3e79e77a4a1756d283b248b6c5930455d98bf24ab523a2b820b28b80feec232"} Jul 22 16:13:15.795222 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.795194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4mrzj" event={"ID":"4102150a-5b4f-4905-b488-b86f1c7e56fb","Type":"ContainerStarted","Data":"a16901f67778da6b7a6fa6dbfb5ea8c79028df4abf5a7ceb7617a55e266fc55c"} Jul 22 16:13:15.795382 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.795242 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4mrzj" event={"ID":"4102150a-5b4f-4905-b488-b86f1c7e56fb","Type":"ContainerStarted","Data":"ab1f6b442775510b57570644a4f935707eef3214a344c0180e4c4a5d5fb06e33"} Jul 22 16:13:15.795382 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.795335 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4mrzj" Jul 22 16:13:15.816341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.816296 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z8drf" podStartSLOduration=66.06869844 podStartE2EDuration="1m10.816282212s" totalImagesPullingTime="4.747583772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:15.816103226 +0000 UTC m=+70.800987245" watchObservedRunningTime="2026-07-22 16:13:15.816282212 +0000 UTC m=+70.801166220" Jul 22 16:13:15.838508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:15.838468 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4mrzj" podStartSLOduration=33.196741139 podStartE2EDuration="36.83845448s" totalImagesPullingTime="3.641713341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:15.837621992 +0000 UTC m=+70.822506002" watchObservedRunningTime="2026-07-22 16:13:15.83845448 +0000 UTC m=+70.823338486" Jul 22 16:13:17.578731 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:17.578695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:13:17.581062 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:17.581036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zmt6t\" (UID: \"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:13:17.856485 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:17.856412 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gktdl\"" Jul 22 16:13:17.864743 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:17.864724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" Jul 22 16:13:17.990363 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:17.990342 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t"] Jul 22 16:13:17.992430 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:17.992404 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854a7df1_e7ed_43a4_99f9_70f2c2bbdb3b.slice/crio-391fab732797c2cb940350aec04fe38028737a5ccd9c87be1b7e7aad7fafab76 WatchSource:0}: Error finding container 391fab732797c2cb940350aec04fe38028737a5ccd9c87be1b7e7aad7fafab76: Status 404 returned error can't find the container with id 391fab732797c2cb940350aec04fe38028737a5ccd9c87be1b7e7aad7fafab76 Jul 22 16:13:18.485487 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.485453 2578 scope.go:117] "RemoveContainer" containerID="3a60038d3874e1466aaca526d440214bae7ab2879708194b38d50f548863527d" Jul 22 16:13:18.806802 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.806778 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:13:18.807176 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.807149 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/1.log" Jul 22 16:13:18.807216 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.807182 2578 generic.go:356] "Generic (PLEG): container finished" podID="a285f20a-93c6-4fd8-9780-49d61469ef5e" containerID="a115644761790c603c8d10feb055263de531bead21ccd06410b602118c279f01" exitCode=255 Jul 22 16:13:18.807275 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.807258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" event={"ID":"a285f20a-93c6-4fd8-9780-49d61469ef5e","Type":"ContainerDied","Data":"a115644761790c603c8d10feb055263de531bead21ccd06410b602118c279f01"} Jul 22 16:13:18.807324 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.807315 2578 scope.go:117] "RemoveContainer" containerID="3a60038d3874e1466aaca526d440214bae7ab2879708194b38d50f548863527d" Jul 22 16:13:18.807669 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.807649 2578 scope.go:117] "RemoveContainer" containerID="a115644761790c603c8d10feb055263de531bead21ccd06410b602118c279f01" Jul 22 16:13:18.807837 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:18.807814 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-smbvj_openshift-console-operator(a285f20a-93c6-4fd8-9780-49d61469ef5e)\"" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" podUID="a285f20a-93c6-4fd8-9780-49d61469ef5e" Jul 22 16:13:18.808576 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:18.808550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" event={"ID":"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b","Type":"ContainerStarted","Data":"391fab732797c2cb940350aec04fe38028737a5ccd9c87be1b7e7aad7fafab76"} Jul 22 16:13:19.046727 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:19.046690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:19.130120 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:19.130020 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:19.814494 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:19.814464 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:13:20.404834 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:20.404794 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:20.485115 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:20.485083 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:20.622011 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:20.621978 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:20.819181 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:20.819150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" event={"ID":"854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b","Type":"ContainerStarted","Data":"808f06f3e0d02a965dc89f930df853a2531289689a2b0de0d70111d44951b317"} Jul 22 16:13:20.860943 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:20.860897 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zmt6t" podStartSLOduration=33.8810563 podStartE2EDuration="35.860882537s" totalImagesPullingTime="1.979826237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:20.860733101 +0000 UTC m=+75.845617108" watchObservedRunningTime="2026-07-22 16:13:20.860882537 +0000 UTC m=+75.845766543" Jul 22 16:13:20.951737 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:20.951713 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:21.910562 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:21.910508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:13:21.912569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:21.912551 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2ab600c-f104-4ad5-a749-f0e5ba6b540a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qnpq\" (UID: \"d2ab600c-f104-4ad5-a749-f0e5ba6b540a\") " pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:13:22.148925 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:22.148897 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88nqz\"" Jul 22 16:13:22.157200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:22.157173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qnpq" Jul 22 16:13:22.291704 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:22.291675 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qnpq"] Jul 22 16:13:22.294701 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:22.294672 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ab600c_f104_4ad5_a749_f0e5ba6b540a.slice/crio-19b25e0c21c95f5b3b761ec05b7243ebdc1bbbce77564f191afee699a5334efa WatchSource:0}: Error finding container 19b25e0c21c95f5b3b761ec05b7243ebdc1bbbce77564f191afee699a5334efa: Status 404 returned error can't find the container with id 19b25e0c21c95f5b3b761ec05b7243ebdc1bbbce77564f191afee699a5334efa Jul 22 16:13:22.828146 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:22.828104 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qnpq" event={"ID":"d2ab600c-f104-4ad5-a749-f0e5ba6b540a","Type":"ContainerStarted","Data":"19b25e0c21c95f5b3b761ec05b7243ebdc1bbbce77564f191afee699a5334efa"} Jul 22 16:13:23.905259 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:23.905223 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:13:23.905703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:23.905291 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:13:23.905703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:23.905631 2578 scope.go:117] "RemoveContainer" containerID="a115644761790c603c8d10feb055263de531bead21ccd06410b602118c279f01" Jul 22 16:13:23.905854 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:23.905827 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-smbvj_openshift-console-operator(a285f20a-93c6-4fd8-9780-49d61469ef5e)\"" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" podUID="a285f20a-93c6-4fd8-9780-49d61469ef5e" Jul 22 16:13:25.801259 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:25.801234 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4mrzj" Jul 22 16:13:26.453687 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.453645 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr"] Jul 22 16:13:26.458179 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.458155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.462356 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.462327 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 16:13:26.462488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.462363 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 16:13:26.462488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.462419 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 16:13:26.462488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.462448 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 16:13:26.482353 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.482329 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr"] Jul 22 16:13:26.549619 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.549592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-tmp\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.549790 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.549632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-klusterlet-config\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.549790 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.549667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcm5w\" (UniqueName: \"kubernetes.io/projected/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-kube-api-access-tcm5w\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.591886 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.591854 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58bd944b45-r42jf"] Jul 22 16:13:26.596932 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.596903 2578 patch_prober.go:28] interesting pod/image-registry-58bd944b45-r42jf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 16:13:26.597055 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.596952 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" podUID="2946fec9-cc76-45ff-ba02-077511935015" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 16:13:26.650303 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.650268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-tmp\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.650303 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.650302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-klusterlet-config\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.650514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.650326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcm5w\" (UniqueName: \"kubernetes.io/projected/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-kube-api-access-tcm5w\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.650725 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.650684 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-tmp\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.653137 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.653112 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-klusterlet-config\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.680476 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.680454 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcm5w\" (UniqueName: \"kubernetes.io/projected/74d00735-4d5f-4c1c-b0e3-0efeab72cb7f-kube-api-access-tcm5w\") pod \"klusterlet-addon-workmgr-654bd7d8db-p7npr\" (UID: \"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.747401 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.747335 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c5575ff86-n8jwq"] Jul 22 16:13:26.755192 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.755163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.762669 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.762632 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:26.769265 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.769238 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:26.811044 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.811024 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5575ff86-n8jwq"] Jul 22 16:13:26.852689 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-bound-sa-token\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.852794 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f81f04-e71d-4c4e-b027-620cde8cf2ba-registry-certificates\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.852861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f81f04-e71d-4c4e-b027-620cde8cf2ba-trusted-ca\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.852861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f81f04-e71d-4c4e-b027-620cde8cf2ba-installation-pull-secrets\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.852954 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f81f04-e71d-4c4e-b027-620cde8cf2ba-image-registry-private-configuration\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.852954 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-registry-tls\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.853034 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f81f04-e71d-4c4e-b027-620cde8cf2ba-ca-trust-extracted\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.853034 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.852990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgt6m\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-kube-api-access-vgt6m\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.918019 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.917982 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:26.954288 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f81f04-e71d-4c4e-b027-620cde8cf2ba-ca-trust-extracted\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954288 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgt6m\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-kube-api-access-vgt6m\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954470 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-bound-sa-token\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954470 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f81f04-e71d-4c4e-b027-620cde8cf2ba-registry-certificates\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954470 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f81f04-e71d-4c4e-b027-620cde8cf2ba-trusted-ca\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954647 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f81f04-e71d-4c4e-b027-620cde8cf2ba-installation-pull-secrets\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954647 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f81f04-e71d-4c4e-b027-620cde8cf2ba-image-registry-private-configuration\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954647 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-registry-tls\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.954802 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.954727 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79f81f04-e71d-4c4e-b027-620cde8cf2ba-ca-trust-extracted\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.955445 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.955378 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79f81f04-e71d-4c4e-b027-620cde8cf2ba-registry-certificates\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.955445 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.955387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79f81f04-e71d-4c4e-b027-620cde8cf2ba-trusted-ca\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.957484 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.957465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79f81f04-e71d-4c4e-b027-620cde8cf2ba-installation-pull-secrets\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.957594 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.957513 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79f81f04-e71d-4c4e-b027-620cde8cf2ba-image-registry-private-configuration\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.957675 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.957651 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-registry-tls\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.973229 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.973128 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgt6m\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-kube-api-access-vgt6m\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:26.980042 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:26.980016 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79f81f04-e71d-4c4e-b027-620cde8cf2ba-bound-sa-token\") pod \"image-registry-5c5575ff86-n8jwq\" (UID: \"79f81f04-e71d-4c4e-b027-620cde8cf2ba\") " pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:27.050499 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.050469 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:27.065807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.065780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:27.156046 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.156018 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:27.174144 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.174122 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr"] Jul 22 16:13:27.183467 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:27.183445 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d00735_4d5f_4c1c_b0e3_0efeab72cb7f.slice/crio-62b8c546b40929388840ecb093ca29dbe214bf5616039c183a98ed0bf1aff888 WatchSource:0}: Error finding container 62b8c546b40929388840ecb093ca29dbe214bf5616039c183a98ed0bf1aff888: Status 404 returned error can't find the container with id 62b8c546b40929388840ecb093ca29dbe214bf5616039c183a98ed0bf1aff888 Jul 22 16:13:27.222917 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.222887 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c5575ff86-n8jwq"] Jul 22 16:13:27.227148 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:27.227126 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f81f04_e71d_4c4e_b027_620cde8cf2ba.slice/crio-8d29870ec914cbf83bf6ba4e5cfeda8913919edee1b4cdda1583c83765ba0d5d WatchSource:0}: Error finding container 8d29870ec914cbf83bf6ba4e5cfeda8913919edee1b4cdda1583c83765ba0d5d: Status 404 returned error can't find the container with id 8d29870ec914cbf83bf6ba4e5cfeda8913919edee1b4cdda1583c83765ba0d5d Jul 22 16:13:27.849182 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.849142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qnpq" event={"ID":"d2ab600c-f104-4ad5-a749-f0e5ba6b540a","Type":"ContainerStarted","Data":"45c750a5cf5e7dc26227fe2bdca5ad5b4cc9832fda975c5d92854ba57fce7402"} Jul 22 16:13:27.850282 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.850255 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" event={"ID":"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f","Type":"ContainerStarted","Data":"62b8c546b40929388840ecb093ca29dbe214bf5616039c183a98ed0bf1aff888"} Jul 22 16:13:27.851899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.851864 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" event={"ID":"79f81f04-e71d-4c4e-b027-620cde8cf2ba","Type":"ContainerStarted","Data":"d4fbd39c51d4cd24f87b9b8956dba062a562e55491b6cbfa7dd86bcab16f2c6c"} Jul 22 16:13:27.851972 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.851913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" event={"ID":"79f81f04-e71d-4c4e-b027-620cde8cf2ba","Type":"ContainerStarted","Data":"8d29870ec914cbf83bf6ba4e5cfeda8913919edee1b4cdda1583c83765ba0d5d"} Jul 22 16:13:27.852152 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.852131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:27.885881 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:27.885838 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" podStartSLOduration=1.885823698 podStartE2EDuration="1.885823698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:27.883760364 +0000 UTC m=+82.868644371" watchObservedRunningTime="2026-07-22 16:13:27.885823698 +0000 UTC m=+82.870707683" Jul 22 16:13:31.733832 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:31.733793 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:31.847379 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:31.847337 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:32.197126 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:32.197088 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:33.146583 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:33.146550 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:33.245891 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:33.245860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:33.876110 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:33.875974 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" event={"ID":"74d00735-4d5f-4c1c-b0e3-0efeab72cb7f","Type":"ContainerStarted","Data":"f1898d771e3a41d49412f53282f568ae4950acc895a6be8c2cead2b8aa373fb3"} Jul 22 16:13:33.876305 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:33.876151 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:33.877966 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:33.877930 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" Jul 22 16:13:33.878436 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:33.878413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qnpq" event={"ID":"d2ab600c-f104-4ad5-a749-f0e5ba6b540a","Type":"ContainerStarted","Data":"a12c6902a14f43aec43f3b38ab24b7311eb31a2c372f14c2c57815ee9b5e1988"} Jul 22 16:13:33.906871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:33.906818 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-654bd7d8db-p7npr" podStartSLOduration=2.076406895 podStartE2EDuration="7.906799649s" totalImagesPullingTime="5.830392754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:33.906494466 +0000 UTC m=+88.891378473" watchObservedRunningTime="2026-07-22 16:13:33.906799649 +0000 UTC m=+88.891683657" Jul 22 16:13:35.138188 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:35.138145 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:35.650286 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:35.650251 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:35.756998 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:35.756962 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:36.596940 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:36.596913 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:36.897248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:36.897121 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qnpq" event={"ID":"d2ab600c-f104-4ad5-a749-f0e5ba6b540a","Type":"ContainerStarted","Data":"2ab643e1eec508557f88c2277682be558b1ef639cef74bd05cd3c007b2da4438"} Jul 22 16:13:36.938985 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:36.938937 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6qnpq" podStartSLOduration=33.747422566 podStartE2EDuration="47.938923318s" totalImagesPullingTime="14.191500752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:12:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:36.937994051 +0000 UTC m=+91.922878047" watchObservedRunningTime="2026-07-22 16:13:36.938923318 +0000 UTC m=+91.923807324" Jul 22 16:13:36.984137 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:36.984100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:37.467223 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.467191 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:37.671922 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.671892 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:37.920409 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.920376 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mtn7x"] Jul 22 16:13:37.922586 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.922571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:37.926316 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.926290 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 16:13:37.926464 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.926391 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 16:13:37.927377 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.927362 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ll44p\"" Jul 22 16:13:37.927446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.927435 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 16:13:37.951646 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.951622 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 16:13:37.999782 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:37.999751 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:38.047280 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-sys\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047408 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-textfile\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047408 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da299873-2d4c-4882-b765-4dae603e0ef8-metrics-client-ca\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047498 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-tls\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047555 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047610 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-root\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047686 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-wtmp\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047742 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkpf\" (UniqueName: \"kubernetes.io/projected/da299873-2d4c-4882-b765-4dae603e0ef8-kube-api-access-7dkpf\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.047779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.047748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148252 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-sys\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-textfile\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da299873-2d4c-4882-b765-4dae603e0ef8-metrics-client-ca\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-tls\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148333 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-sys\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-root\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148436 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-root\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-wtmp\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkpf\" (UniqueName: \"kubernetes.io/projected/da299873-2d4c-4882-b765-4dae603e0ef8-kube-api-access-7dkpf\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148618 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-textfile\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.148828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-wtmp\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.149117 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.148928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da299873-2d4c-4882-b765-4dae603e0ef8-metrics-client-ca\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.149117 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.149092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.150710 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.150689 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.151053 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.151029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da299873-2d4c-4882-b765-4dae603e0ef8-node-exporter-tls\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.178192 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.178134 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkpf\" (UniqueName: \"kubernetes.io/projected/da299873-2d4c-4882-b765-4dae603e0ef8-kube-api-access-7dkpf\") pod \"node-exporter-mtn7x\" (UID: \"da299873-2d4c-4882-b765-4dae603e0ef8\") " pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.205844 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.205819 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:38.231407 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.231375 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mtn7x" Jul 22 16:13:38.240883 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:38.240858 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda299873_2d4c_4882_b765_4dae603e0ef8.slice/crio-21674642a8bc89762aaefc6e3b4c9394a3874dd5a3b926e02d0a483a5ae56bd9 WatchSource:0}: Error finding container 21674642a8bc89762aaefc6e3b4c9394a3874dd5a3b926e02d0a483a5ae56bd9: Status 404 returned error can't find the container with id 21674642a8bc89762aaefc6e3b4c9394a3874dd5a3b926e02d0a483a5ae56bd9 Jul 22 16:13:38.361940 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.361903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:38.563768 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.563734 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:38.769628 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.769596 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:13:38.772220 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.772196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.786601 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.786579 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 16:13:38.786804 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.786640 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 16:13:38.786804 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.786722 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 16:13:38.786804 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.786731 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 16:13:38.786981 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.786863 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 16:13:38.787063 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.786984 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 16:13:38.787063 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.787012 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fkx9k\"" Jul 22 16:13:38.787200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.787018 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 16:13:38.787291 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.787276 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 16:13:38.798137 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.798112 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:13:38.799256 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.799233 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 16:13:38.853423 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-web-config\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-config-out\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853647 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-config-volume\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853713 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27d7\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-kube-api-access-n27d7\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.853779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.854000 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.853788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.905205 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.905162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtn7x" event={"ID":"da299873-2d4c-4882-b765-4dae603e0ef8","Type":"ContainerStarted","Data":"21674642a8bc89762aaefc6e3b4c9394a3874dd5a3b926e02d0a483a5ae56bd9"} Jul 22 16:13:38.954589 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.954550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.954760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.954612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.954760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.954658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.954760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.954677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.954760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.954695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-web-config\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.954760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.954712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-config-out\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955037 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.954944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955037 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.955003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955144 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.955043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955144 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.955077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-config-volume\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955144 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.955116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27d7\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-kube-api-access-n27d7\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955296 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.955153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955296 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.955176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.955860 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.955554 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.956550 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.956341 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.957275 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.956926 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.958350 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.958324 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.958550 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.958507 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.958773 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.958743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.958872 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.958835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-web-config\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.959057 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.959034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-config-out\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.959347 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.959324 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.959999 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.959977 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.960884 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.960835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-config-volume\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.961099 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.961082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:38.973674 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:38.973643 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27d7\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-kube-api-access-n27d7\") pod \"alertmanager-main-0\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:39.005295 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.005262 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:39.083418 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.083384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:13:39.128465 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.128424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:39.264759 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.262455 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:13:39.268662 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:39.268508 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92baafac_4291_428f_a2d9_60f286971e35.slice/crio-8cda558e825c30eee0e64564566040b07ca0246e7215cf05dd1552159016f7fe WatchSource:0}: Error finding container 8cda558e825c30eee0e64564566040b07ca0246e7215cf05dd1552159016f7fe: Status 404 returned error can't find the container with id 8cda558e825c30eee0e64564566040b07ca0246e7215cf05dd1552159016f7fe Jul 22 16:13:39.485662 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.485580 2578 scope.go:117] "RemoveContainer" containerID="a115644761790c603c8d10feb055263de531bead21ccd06410b602118c279f01" Jul 22 16:13:39.909774 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.909738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerStarted","Data":"8cda558e825c30eee0e64564566040b07ca0246e7215cf05dd1552159016f7fe"} Jul 22 16:13:39.911662 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.911640 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:13:39.911764 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.911751 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" event={"ID":"a285f20a-93c6-4fd8-9780-49d61469ef5e","Type":"ContainerStarted","Data":"e9060e8e421436ae7c40cfe0e34fa71db8b6c364fcc122c55ade672e6ef1d045"} Jul 22 16:13:39.912085 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.912061 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:13:39.913658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.913629 2578 generic.go:356] "Generic (PLEG): container finished" podID="da299873-2d4c-4882-b765-4dae603e0ef8" containerID="7e201eeccd88a486648600c440ab0d16e40e23c5a15c97becbd4549bce99f39e" exitCode=0 Jul 22 16:13:39.913762 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.913674 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtn7x" event={"ID":"da299873-2d4c-4882-b765-4dae603e0ef8","Type":"ContainerDied","Data":"7e201eeccd88a486648600c440ab0d16e40e23c5a15c97becbd4549bce99f39e"} Jul 22 16:13:39.975922 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:39.975863 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" podStartSLOduration=44.389361518 podStartE2EDuration="46.975843808s" totalImagesPullingTime="2.58648229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:39.951968884 +0000 UTC m=+94.936852891" watchObservedRunningTime="2026-07-22 16:13:39.975843808 +0000 UTC m=+94.960727816" Jul 22 16:13:40.080736 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.080706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-smbvj" Jul 22 16:13:40.297512 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.297480 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-bkvbz"] Jul 22 16:13:40.300106 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.300083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-bkvbz" Jul 22 16:13:40.305564 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.305541 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 16:13:40.305883 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.305860 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 16:13:40.308625 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.308606 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g44hp\"" Jul 22 16:13:40.341281 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.341257 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-bkvbz"] Jul 22 16:13:40.365085 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.365060 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:40.468912 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.468885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xkx\" (UniqueName: \"kubernetes.io/projected/61042d59-56c3-452d-829b-b4459972c1a5-kube-api-access-q4xkx\") pod \"downloads-5bc67dcd46-bkvbz\" (UID: \"61042d59-56c3-452d-829b-b4459972c1a5\") " pod="openshift-console/downloads-5bc67dcd46-bkvbz" Jul 22 16:13:40.498464 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.498410 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:40.569706 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.569678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xkx\" (UniqueName: \"kubernetes.io/projected/61042d59-56c3-452d-829b-b4459972c1a5-kube-api-access-q4xkx\") pod \"downloads-5bc67dcd46-bkvbz\" (UID: \"61042d59-56c3-452d-829b-b4459972c1a5\") " pod="openshift-console/downloads-5bc67dcd46-bkvbz" Jul 22 16:13:40.595743 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.595718 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xkx\" (UniqueName: \"kubernetes.io/projected/61042d59-56c3-452d-829b-b4459972c1a5-kube-api-access-q4xkx\") pod \"downloads-5bc67dcd46-bkvbz\" (UID: \"61042d59-56c3-452d-829b-b4459972c1a5\") " pod="openshift-console/downloads-5bc67dcd46-bkvbz" Jul 22 16:13:40.614647 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.614623 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-bkvbz" Jul 22 16:13:40.687377 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.687348 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:40.753380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.753354 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-bkvbz"] Jul 22 16:13:40.757912 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:40.757890 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61042d59_56c3_452d_829b_b4459972c1a5.slice/crio-531b2458bc9c13556f32604d18f6bd1460c043379a8b403708f167db2cc4ff29 WatchSource:0}: Error finding container 531b2458bc9c13556f32604d18f6bd1460c043379a8b403708f167db2cc4ff29: Status 404 returned error can't find the container with id 531b2458bc9c13556f32604d18f6bd1460c043379a8b403708f167db2cc4ff29 Jul 22 16:13:40.809663 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.809612 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk"] Jul 22 16:13:40.812060 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.812042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.816021 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:40.815996 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"thanos-querier-749bf4c8d6-q2xhk\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" podUID="0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.816411 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:40.816390 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy-metrics\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" Jul 22 16:13:40.816749 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:40.816721 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-dockercfg-5txgd\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5txgd\"" type="*v1.Secret" Jul 22 16:13:40.816886 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:40.816863 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-grpc-tls-4lvi8cs9pp34n\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4lvi8cs9pp34n\"" type="*v1.Secret" Jul 22 16:13:40.816993 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:40.816972 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy-rules\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" Jul 22 16:13:40.823187 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:40.823157 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-tls\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" Jul 22 16:13:40.824966 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:40.824946 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"thanos-querier-kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" Jul 22 16:13:40.825038 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.824968 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 16:13:40.834561 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.834537 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk"] Jul 22 16:13:40.918978 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.918949 2578 generic.go:356] "Generic (PLEG): container finished" podID="92baafac-4291-428f-a2d9-60f286971e35" containerID="68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf" exitCode=0 Jul 22 16:13:40.919390 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.919031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf"} Jul 22 16:13:40.921589 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.921564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtn7x" event={"ID":"da299873-2d4c-4882-b765-4dae603e0ef8","Type":"ContainerStarted","Data":"f150f1213ab113f07e498b6c07b7c1a5959165d2616ed0441b45b402c57cfd0e"} Jul 22 16:13:40.921699 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.921612 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtn7x" event={"ID":"da299873-2d4c-4882-b765-4dae603e0ef8","Type":"ContainerStarted","Data":"f4ebaf52c7c8acc6691b896e693fb7a9c6aafd30af919a6b89fb1bd7b87c9f9f"} Jul 22 16:13:40.922940 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.922921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-bkvbz" event={"ID":"61042d59-56c3-452d-829b-b4459972c1a5","Type":"ContainerStarted","Data":"531b2458bc9c13556f32604d18f6bd1460c043379a8b403708f167db2cc4ff29"} Jul 22 16:13:40.972625 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9jc\" (UniqueName: \"kubernetes.io/projected/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-kube-api-access-2h9jc\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.972745 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.972745 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-tls\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.972745 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-metrics-client-ca\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.972745 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.972887 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.972887 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.972887 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.972883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-grpc-tls\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:40.992278 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:40.992248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:41.073964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.073887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9jc\" (UniqueName: \"kubernetes.io/projected/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-kube-api-access-2h9jc\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.073964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.073934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.074157 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.074000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-tls\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.074557 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.074254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-metrics-client-ca\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.074557 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.074322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.074760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.074740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.074848 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.074833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.074957 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.074884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-grpc-tls\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.075132 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.075115 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-metrics-client-ca\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.076586 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.076561 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.093718 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.093670 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mtn7x" podStartSLOduration=3.14063973 podStartE2EDuration="4.09365511s" totalImagesPullingTime="882.467065ms" totalInitContainerRuntime="70.548315ms" isStatefulPod=true podCreationTimestamp="2026-07-22 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:41.093136507 +0000 UTC m=+96.078020534" watchObservedRunningTime="2026-07-22 16:13:41.09365511 +0000 UTC m=+96.078539117" Jul 22 16:13:41.095714 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.095695 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9jc\" (UniqueName: \"kubernetes.io/projected/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-kube-api-access-2h9jc\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.157861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.157825 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:41.353937 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.353846 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:41.605661 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.605497 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:41.777744 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.777715 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gpxss" Jul 22 16:13:41.888150 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.888081 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4lvi8cs9pp34n\"" Jul 22 16:13:41.892065 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.892034 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 16:13:41.898117 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.898089 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-grpc-tls\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.898223 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.898130 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:41.947245 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.947221 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5txgd\"" Jul 22 16:13:41.990172 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:41.990140 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:42.000145 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.000122 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx"] Jul 22 16:13:42.003267 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.003249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.006190 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.006170 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 16:13:42.006714 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.006553 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zndgg\"" Jul 22 16:13:42.020741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.020719 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx"] Jul 22 16:13:42.074178 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:42.074152 2578 secret.go:189] Couldn't get secret openshift-monitoring/thanos-querier-tls: failed to sync secret cache: timed out waiting for the condition Jul 22 16:13:42.074305 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:42.074230 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-tls podName:0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95 nodeName:}" failed. No retries permitted until 2026-07-22 16:13:42.574209865 +0000 UTC m=+97.559093849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-thanos-querier-tls" (UniqueName: "kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-tls") pod "thanos-querier-749bf4c8d6-q2xhk" (UID: "0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95") : failed to sync secret cache: timed out waiting for the condition Jul 22 16:13:42.075246 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:42.075226 2578 secret.go:189] Couldn't get secret openshift-monitoring/thanos-querier-kube-rbac-proxy: failed to sync secret cache: timed out waiting for the condition Jul 22 16:13:42.075246 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:42.075238 2578 secret.go:189] Couldn't get secret openshift-monitoring/thanos-querier-kube-rbac-proxy-rules: failed to sync secret cache: timed out waiting for the condition Jul 22 16:13:42.075380 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:42.075292 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy podName:0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95 nodeName:}" failed. No retries permitted until 2026-07-22 16:13:42.575273889 +0000 UTC m=+97.560157877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-thanos-querier-kube-rbac-proxy" (UniqueName: "kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy") pod "thanos-querier-749bf4c8d6-q2xhk" (UID: "0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95") : failed to sync secret cache: timed out waiting for the condition Jul 22 16:13:42.075380 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:42.075309 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-rules podName:0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95 nodeName:}" failed. No retries permitted until 2026-07-22 16:13:42.575302124 +0000 UTC m=+97.560186108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-thanos-querier-kube-rbac-proxy-rules" (UniqueName: "kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-rules") pod "thanos-querier-749bf4c8d6-q2xhk" (UID: "0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95") : failed to sync secret cache: timed out waiting for the condition Jul 22 16:13:42.082882 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.082815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34fdaec6-98cb-4b19-9d76-e07618935db2-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-bjmbx\" (UID: \"34fdaec6-98cb-4b19-9d76-e07618935db2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.082882 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.082858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rn4\" (UniqueName: \"kubernetes.io/projected/34fdaec6-98cb-4b19-9d76-e07618935db2-kube-api-access-66rn4\") pod \"monitoring-plugin-7b7b7bc8bf-bjmbx\" (UID: \"34fdaec6-98cb-4b19-9d76-e07618935db2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.096411 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.096381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:42.183782 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.183701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34fdaec6-98cb-4b19-9d76-e07618935db2-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-bjmbx\" (UID: \"34fdaec6-98cb-4b19-9d76-e07618935db2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.183782 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.183748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rn4\" (UniqueName: \"kubernetes.io/projected/34fdaec6-98cb-4b19-9d76-e07618935db2-kube-api-access-66rn4\") pod \"monitoring-plugin-7b7b7bc8bf-bjmbx\" (UID: \"34fdaec6-98cb-4b19-9d76-e07618935db2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.183971 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.183878 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 16:13:42.186757 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.186734 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34fdaec6-98cb-4b19-9d76-e07618935db2-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-bjmbx\" (UID: \"34fdaec6-98cb-4b19-9d76-e07618935db2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.204300 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.204276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rn4\" (UniqueName: \"kubernetes.io/projected/34fdaec6-98cb-4b19-9d76-e07618935db2-kube-api-access-66rn4\") pod \"monitoring-plugin-7b7b7bc8bf-bjmbx\" (UID: \"34fdaec6-98cb-4b19-9d76-e07618935db2\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.289103 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.288973 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:42.315280 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.315244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:42.365350 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.365313 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 16:13:42.409727 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.409701 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 16:13:42.425304 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.425233 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:42.548407 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.548359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:42.588360 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.588331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-tls\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:42.588501 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.588415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:42.588501 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.588450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:42.591416 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.591395 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:42.591511 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.591489 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-tls\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:42.591511 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.591497 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-749bf4c8d6-q2xhk\" (UID: \"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95\") " pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:42.620629 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.620608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:42.754615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.754342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:42.805610 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.805557 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx"] Jul 22 16:13:42.811704 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:42.811680 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fdaec6_98cb_4b19_9d76_e07618935db2.slice/crio-dc745731b5317e2771be379c036d7e46abeec57cdff45a04d542ecbaeed14a4d WatchSource:0}: Error finding container dc745731b5317e2771be379c036d7e46abeec57cdff45a04d542ecbaeed14a4d: Status 404 returned error can't find the container with id dc745731b5317e2771be379c036d7e46abeec57cdff45a04d542ecbaeed14a4d Jul 22 16:13:42.828998 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.828960 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk"] Jul 22 16:13:42.833705 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:42.833673 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e76ee68_2c2d_4ea7_a505_09d3dc7c1d95.slice/crio-caa790e3417838a75c1aab0ffe955c987b99215573b485b3740312f2237922e7 WatchSource:0}: Error finding container caa790e3417838a75c1aab0ffe955c987b99215573b485b3740312f2237922e7: Status 404 returned error can't find the container with id caa790e3417838a75c1aab0ffe955c987b99215573b485b3740312f2237922e7 Jul 22 16:13:42.870309 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.870279 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:42.944532 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.944488 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerStarted","Data":"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e"} Jul 22 16:13:42.944644 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.944632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerStarted","Data":"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281"} Jul 22 16:13:42.944705 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.944675 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerStarted","Data":"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad"} Jul 22 16:13:42.945750 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.945722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" event={"ID":"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95","Type":"ContainerStarted","Data":"caa790e3417838a75c1aab0ffe955c987b99215573b485b3740312f2237922e7"} Jul 22 16:13:42.947176 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.947153 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" event={"ID":"34fdaec6-98cb-4b19-9d76-e07618935db2","Type":"ContainerStarted","Data":"dc745731b5317e2771be379c036d7e46abeec57cdff45a04d542ecbaeed14a4d"} Jul 22 16:13:42.979149 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:42.979087 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:43.958813 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:43.958726 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerStarted","Data":"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d"} Jul 22 16:13:43.958971 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:43.958825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerStarted","Data":"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c"} Jul 22 16:13:44.001849 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:44.001751 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:44.246769 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:44.246682 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:44.339181 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:44.339126 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:45.396004 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:45.395970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:45.520376 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:45.520340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:46.975501 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.975415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" event={"ID":"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95","Type":"ContainerStarted","Data":"f5fa5e0e9ddfc35bbb7b3aba0da6dfefec795b0a1c3a56137357465660da2711"} Jul 22 16:13:46.975982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.975551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" event={"ID":"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95","Type":"ContainerStarted","Data":"fa1989f6e6031905d206228977518e416877eb8c8b1bfca385f03d2906ebd5e7"} Jul 22 16:13:46.975982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.975600 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:46.975982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.975650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" event={"ID":"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95","Type":"ContainerStarted","Data":"82d456b2ff38833b4770f08d67c91f04a007f43fa626d19ac758b7d8b8e40987"} Jul 22 16:13:46.975982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.975691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" event={"ID":"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95","Type":"ContainerStarted","Data":"e55e3d10aeb17500d23af7ece1b4b8d27921bca5d238f4f00abfd6588277b735"} Jul 22 16:13:46.975982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.975731 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" event={"ID":"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95","Type":"ContainerStarted","Data":"ba920d3f79c1983b036ecce83cace117e6b04ee2b6ef5e1abbe72f583d88aee7"} Jul 22 16:13:46.975982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.975769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" event={"ID":"0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95","Type":"ContainerStarted","Data":"0afb3ff0e679019737dadb9942e83cb16017359011ea99ca7e453201104e1306"} Jul 22 16:13:46.977563 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.977507 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" event={"ID":"34fdaec6-98cb-4b19-9d76-e07618935db2","Type":"ContainerStarted","Data":"b5f3d2b7933c832c2bd75ec732db578fd55cd4d3fa0b805ce3234c271b5f21f5"} Jul 22 16:13:46.977681 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.977579 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:46.981773 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.981742 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerStarted","Data":"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2"} Jul 22 16:13:46.984079 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:46.984055 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" Jul 22 16:13:47.014926 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:47.014883 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" podStartSLOduration=3.861579566 podStartE2EDuration="7.014863688s" totalImagesPullingTime="3.153284122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:47.014286003 +0000 UTC m=+101.999170019" watchObservedRunningTime="2026-07-22 16:13:47.014863688 +0000 UTC m=+101.999747694" Jul 22 16:13:47.066224 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:47.066180 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.121356729 podStartE2EDuration="9.066163812s" totalImagesPullingTime="5.924722672s" totalInitContainerRuntime="20.084411ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:47.063164396 +0000 UTC m=+102.048048402" watchObservedRunningTime="2026-07-22 16:13:47.066163812 +0000 UTC m=+102.051047819" Jul 22 16:13:47.931786 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:47.931750 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:48.060633 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:48.060581 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:48.861351 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:48.861321 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c5575ff86-n8jwq" Jul 22 16:13:48.891407 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:48.891356 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-bjmbx" podStartSLOduration=4.723550815 podStartE2EDuration="7.891339676s" totalImagesPullingTime="3.167788861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:13:47.093839153 +0000 UTC m=+102.078723160" watchObservedRunningTime="2026-07-22 16:13:48.891339676 +0000 UTC m=+103.876223693" Jul 22 16:13:49.110942 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.110909 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:49.215510 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.215422 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:49.324135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.324107 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:49.427809 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.427776 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:49.841222 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.841188 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5664d4dcb6-6wxkb"] Jul 22 16:13:49.843410 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.843389 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.847655 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.847632 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 16:13:49.847922 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.847898 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 16:13:49.848195 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.848177 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 16:13:49.848884 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.848862 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gb9f8\"" Jul 22 16:13:49.848997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.848868 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 16:13:49.848997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.848867 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 16:13:49.859988 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.859947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-config\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.860132 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.860027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snnqw\" (UniqueName: \"kubernetes.io/projected/a27d0577-d0db-4d5a-ab75-61ed9cf86105-kube-api-access-snnqw\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.860132 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.860076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-oauth-config\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.860297 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.860142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-service-ca\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.860297 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.860183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-oauth-serving-cert\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.860297 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.860224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-serving-cert\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.862898 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.862875 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5664d4dcb6-6wxkb"] Jul 22 16:13:49.961104 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.961072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-oauth-config\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.961233 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.961131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-service-ca\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.961233 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.961170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-oauth-serving-cert\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.961233 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.961207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-serving-cert\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.961423 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.961400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-config\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.961505 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.961489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snnqw\" (UniqueName: \"kubernetes.io/projected/a27d0577-d0db-4d5a-ab75-61ed9cf86105-kube-api-access-snnqw\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.962011 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.961990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-oauth-serving-cert\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.962011 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.962005 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-service-ca\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.962157 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.962076 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-config\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.963629 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.963607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-oauth-config\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.963733 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.963660 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-serving-cert\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:49.972755 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:49.972737 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snnqw\" (UniqueName: \"kubernetes.io/projected/a27d0577-d0db-4d5a-ab75-61ed9cf86105-kube-api-access-snnqw\") pod \"console-5664d4dcb6-6wxkb\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:50.155143 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:50.155066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:13:50.310124 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:50.310101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5664d4dcb6-6wxkb"] Jul 22 16:13:50.312860 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:13:50.312822 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27d0577_d0db_4d5a_ab75_61ed9cf86105.slice/crio-df5b4b529f35e8bb232f1728f10d1ece366bc9998e0a3e80f64163d5ebc7b63c WatchSource:0}: Error finding container df5b4b529f35e8bb232f1728f10d1ece366bc9998e0a3e80f64163d5ebc7b63c: Status 404 returned error can't find the container with id df5b4b529f35e8bb232f1728f10d1ece366bc9998e0a3e80f64163d5ebc7b63c Jul 22 16:13:51.000384 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.000337 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5664d4dcb6-6wxkb" event={"ID":"a27d0577-d0db-4d5a-ab75-61ed9cf86105","Type":"ContainerStarted","Data":"df5b4b529f35e8bb232f1728f10d1ece366bc9998e0a3e80f64163d5ebc7b63c"} Jul 22 16:13:51.249849 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.249752 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:51.352453 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.352350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:51.616075 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.615983 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" podUID="2946fec9-cc76-45ff-ba02-077511935015" containerName="registry" containerID="cri-o://3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2" gracePeriod=30 Jul 22 16:13:51.888751 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.888725 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:51.982642 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982486 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-bound-sa-token\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.982642 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982556 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2946fec9-cc76-45ff-ba02-077511935015-ca-trust-extracted\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.982642 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982588 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-image-registry-private-configuration\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.982642 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982619 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-registry-certificates\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.982955 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982660 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtgvp\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-kube-api-access-jtgvp\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.982955 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982700 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-trusted-ca\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.982955 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982753 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-installation-pull-secrets\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.982955 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.982796 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") pod \"2946fec9-cc76-45ff-ba02-077511935015\" (UID: \"2946fec9-cc76-45ff-ba02-077511935015\") " Jul 22 16:13:51.984272 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.984034 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:13:51.984272 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.984239 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:13:51.987877 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.987846 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:13:51.988790 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.988762 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:13:51.989099 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.989057 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:13:51.989826 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.989802 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-kube-api-access-jtgvp" (OuterVolumeSpecName: "kube-api-access-jtgvp") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "kube-api-access-jtgvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:13:51.990848 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.990814 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:13:51.994907 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:51.994879 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2946fec9-cc76-45ff-ba02-077511935015-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2946fec9-cc76-45ff-ba02-077511935015" (UID: "2946fec9-cc76-45ff-ba02-077511935015"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:13:52.006952 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.006921 2578 generic.go:356] "Generic (PLEG): container finished" podID="2946fec9-cc76-45ff-ba02-077511935015" containerID="3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2" exitCode=0 Jul 22 16:13:52.007091 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.006974 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" Jul 22 16:13:52.007091 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.006984 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" event={"ID":"2946fec9-cc76-45ff-ba02-077511935015","Type":"ContainerDied","Data":"3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2"} Jul 22 16:13:52.007091 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.007029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bd944b45-r42jf" event={"ID":"2946fec9-cc76-45ff-ba02-077511935015","Type":"ContainerDied","Data":"f1a3ef3baf0161e1a9def91194f3211426e347fdbf4acbde916acf764033df49"} Jul 22 16:13:52.007091 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.007056 2578 scope.go:117] "RemoveContainer" containerID="3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2" Jul 22 16:13:52.018376 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.018257 2578 scope.go:117] "RemoveContainer" containerID="3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2" Jul 22 16:13:52.019102 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:13:52.018938 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2\": container with ID starting with 3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2 not found: ID does not exist" containerID="3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2" Jul 22 16:13:52.019102 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.018978 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2"} err="failed to get container status \"3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2\": rpc error: code = NotFound desc = could not find container \"3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2\": container with ID starting with 3c54668670dd0bda90471d4009aa9f52caca391f809053794935936c922b62f2 not found: ID does not exist" Jul 22 16:13:52.038466 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.038434 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58bd944b45-r42jf"] Jul 22 16:13:52.043320 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.043293 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58bd944b45-r42jf"] Jul 22 16:13:52.084333 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084290 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-registry-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.084333 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084322 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-bound-sa-token\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.084333 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084334 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2946fec9-cc76-45ff-ba02-077511935015-ca-trust-extracted\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.084645 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084348 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-image-registry-private-configuration\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.084645 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084363 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-registry-certificates\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.084645 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084375 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtgvp\" (UniqueName: \"kubernetes.io/projected/2946fec9-cc76-45ff-ba02-077511935015-kube-api-access-jtgvp\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.084645 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084388 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2946fec9-cc76-45ff-ba02-077511935015-trusted-ca\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.084645 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.084401 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2946fec9-cc76-45ff-ba02-077511935015-installation-pull-secrets\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:13:52.995750 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:52.995704 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-749bf4c8d6-q2xhk" Jul 22 16:13:53.491418 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:53.491378 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2946fec9-cc76-45ff-ba02-077511935015" path="/var/lib/kubelet/pods/2946fec9-cc76-45ff-ba02-077511935015/volumes" Jul 22 16:13:54.827196 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:54.827163 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:54.927383 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:54.927342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:55.282348 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:55.282315 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:55.401162 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:55.401112 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:56.012701 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:56.012665 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:56.106896 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:56.106859 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:58.658346 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:58.658309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:13:58.742691 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:13:58.742643 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:00.488060 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:00.488023 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:00.631379 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:00.631351 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:00.747748 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:00.747672 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:01.048856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:01.048825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-bkvbz" event={"ID":"61042d59-56c3-452d-829b-b4459972c1a5","Type":"ContainerStarted","Data":"e4b2041de967b39b206db6441198bb989ceacf23fb5600efb668a9d4ada2fc62"} Jul 22 16:14:01.048997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:01.048894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-bkvbz" Jul 22 16:14:01.050024 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:01.049999 2578 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-bkvbz container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.18:8080/\": dial tcp 10.134.0.18:8080: connect: connection refused" start-of-body= Jul 22 16:14:01.050150 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:01.050036 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-bkvbz" podUID="61042d59-56c3-452d-829b-b4459972c1a5" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.18:8080/\": dial tcp 10.134.0.18:8080: connect: connection refused" Jul 22 16:14:01.050847 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:01.050824 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5664d4dcb6-6wxkb" event={"ID":"a27d0577-d0db-4d5a-ab75-61ed9cf86105","Type":"ContainerStarted","Data":"3b3512a7f8a291923c56166df6d8ca7701103abf65f737aa03458d2965feb14d"} Jul 22 16:14:01.114864 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:01.114820 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5664d4dcb6-6wxkb" podStartSLOduration=1.569339424 podStartE2EDuration="12.114803748s" totalImagesPullingTime="10.545464324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:14:01.113568162 +0000 UTC m=+116.098452169" watchObservedRunningTime="2026-07-22 16:14:01.114803748 +0000 UTC m=+116.099687755" Jul 22 16:14:01.115393 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:01.115367 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-bkvbz" podStartSLOduration=0.96449535 podStartE2EDuration="21.115360424s" totalImagesPullingTime="20.150865074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:14:01.086834203 +0000 UTC m=+116.071718208" watchObservedRunningTime="2026-07-22 16:14:01.115360424 +0000 UTC m=+116.100244432" Jul 22 16:14:02.074314 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:02.074276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-bkvbz" Jul 22 16:14:04.048418 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:04.048376 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:04.125924 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:04.125886 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:07.535274 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:07.535231 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:07.715828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:07.715789 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:07.962727 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:07.962641 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:08.093635 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:08.093600 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:08.964735 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:08.964701 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5664d4dcb6-6wxkb"] Jul 22 16:14:09.286847 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:09.286804 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:09.389065 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:09.389026 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:10.122710 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:10.122677 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:10.155199 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:10.155171 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:14:10.226737 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:10.226709 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:10.323117 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:10.323074 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:10.412753 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:10.412671 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:14.278641 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:14.278603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:14:14.280863 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:14.280844 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06184455-9451-417f-9a46-2009651babfc-metrics-certs\") pod \"network-metrics-daemon-mwwp5\" (UID: \"06184455-9451-417f-9a46-2009651babfc\") " pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:14:14.399337 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:14.399312 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-df68t\"" Jul 22 16:14:14.406760 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:14.406741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwwp5" Jul 22 16:14:14.626345 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:14.626321 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwwp5"] Jul 22 16:14:14.628688 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:14:14.628655 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06184455_9451_417f_9a46_2009651babfc.slice/crio-4099644a4c9139a5261e2721ed0fb3993da67a4fe89ee9ff1ab18e3a911d6944 WatchSource:0}: Error finding container 4099644a4c9139a5261e2721ed0fb3993da67a4fe89ee9ff1ab18e3a911d6944: Status 404 returned error can't find the container with id 4099644a4c9139a5261e2721ed0fb3993da67a4fe89ee9ff1ab18e3a911d6944 Jul 22 16:14:15.106532 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:15.106496 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwwp5" event={"ID":"06184455-9451-417f-9a46-2009651babfc","Type":"ContainerStarted","Data":"4099644a4c9139a5261e2721ed0fb3993da67a4fe89ee9ff1ab18e3a911d6944"} Jul 22 16:14:17.117690 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:17.117651 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwwp5" event={"ID":"06184455-9451-417f-9a46-2009651babfc","Type":"ContainerStarted","Data":"98d829e7b0d734b04e85d878e972e21b110bfc7e6883848794411546ec201204"} Jul 22 16:14:17.118051 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:17.117708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwwp5" event={"ID":"06184455-9451-417f-9a46-2009651babfc","Type":"ContainerStarted","Data":"03da6041ea73eceb519267b8c1890be033148df19119ed847d40a2d471b56e8d"} Jul 22 16:14:17.149703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:17.149657 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mwwp5" podStartSLOduration=130.711422948 podStartE2EDuration="2m12.149643448s" totalImagesPullingTime="1.4382205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:14:17.149145701 +0000 UTC m=+132.134029708" watchObservedRunningTime="2026-07-22 16:14:17.149643448 +0000 UTC m=+132.134527454" Jul 22 16:14:17.360997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:17.360969 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:17.493306 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:17.493217 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:19.065041 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:19.064998 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:19.147726 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:19.147693 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:24.202665 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:24.202630 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:34.004051 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.004005 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5664d4dcb6-6wxkb" podUID="a27d0577-d0db-4d5a-ab75-61ed9cf86105" containerName="console" containerID="cri-o://3b3512a7f8a291923c56166df6d8ca7701103abf65f737aa03458d2965feb14d" gracePeriod=15 Jul 22 16:14:34.187883 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.187850 2578 generic.go:356] "Generic (PLEG): container finished" podID="a27d0577-d0db-4d5a-ab75-61ed9cf86105" containerID="3b3512a7f8a291923c56166df6d8ca7701103abf65f737aa03458d2965feb14d" exitCode=0 Jul 22 16:14:34.188003 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.187913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5664d4dcb6-6wxkb" event={"ID":"a27d0577-d0db-4d5a-ab75-61ed9cf86105","Type":"ContainerDied","Data":"3b3512a7f8a291923c56166df6d8ca7701103abf65f737aa03458d2965feb14d"} Jul 22 16:14:34.283488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.283460 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:14:34.344663 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.344623 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-oauth-serving-cert\") pod \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " Jul 22 16:14:34.344821 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.344692 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-oauth-config\") pod \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " Jul 22 16:14:34.344821 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.344716 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-service-ca\") pod \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " Jul 22 16:14:34.344821 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.344770 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-serving-cert\") pod \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " Jul 22 16:14:34.344821 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.344799 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-config\") pod \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " Jul 22 16:14:34.345036 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.344827 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snnqw\" (UniqueName: \"kubernetes.io/projected/a27d0577-d0db-4d5a-ab75-61ed9cf86105-kube-api-access-snnqw\") pod \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\" (UID: \"a27d0577-d0db-4d5a-ab75-61ed9cf86105\") " Jul 22 16:14:34.345160 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.345137 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a27d0577-d0db-4d5a-ab75-61ed9cf86105" (UID: "a27d0577-d0db-4d5a-ab75-61ed9cf86105"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:14:34.345216 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.345185 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-service-ca" (OuterVolumeSpecName: "service-ca") pod "a27d0577-d0db-4d5a-ab75-61ed9cf86105" (UID: "a27d0577-d0db-4d5a-ab75-61ed9cf86105"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:14:34.345281 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.345224 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-config" (OuterVolumeSpecName: "console-config") pod "a27d0577-d0db-4d5a-ab75-61ed9cf86105" (UID: "a27d0577-d0db-4d5a-ab75-61ed9cf86105"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:14:34.347006 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.346976 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a27d0577-d0db-4d5a-ab75-61ed9cf86105" (UID: "a27d0577-d0db-4d5a-ab75-61ed9cf86105"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:34.347112 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.347019 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27d0577-d0db-4d5a-ab75-61ed9cf86105-kube-api-access-snnqw" (OuterVolumeSpecName: "kube-api-access-snnqw") pod "a27d0577-d0db-4d5a-ab75-61ed9cf86105" (UID: "a27d0577-d0db-4d5a-ab75-61ed9cf86105"). InnerVolumeSpecName "kube-api-access-snnqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:14:34.347112 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.347026 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a27d0577-d0db-4d5a-ab75-61ed9cf86105" (UID: "a27d0577-d0db-4d5a-ab75-61ed9cf86105"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:34.446161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.446130 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-oauth-config\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:34.446161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.446154 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-service-ca\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:34.446161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.446164 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-serving-cert\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:34.446355 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.446173 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-console-config\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:34.446355 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.446182 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snnqw\" (UniqueName: \"kubernetes.io/projected/a27d0577-d0db-4d5a-ab75-61ed9cf86105-kube-api-access-snnqw\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:34.446355 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:34.446190 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27d0577-d0db-4d5a-ab75-61ed9cf86105-oauth-serving-cert\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:35.192715 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:35.192676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5664d4dcb6-6wxkb" event={"ID":"a27d0577-d0db-4d5a-ab75-61ed9cf86105","Type":"ContainerDied","Data":"df5b4b529f35e8bb232f1728f10d1ece366bc9998e0a3e80f64163d5ebc7b63c"} Jul 22 16:14:35.192715 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:35.192708 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5664d4dcb6-6wxkb" Jul 22 16:14:35.193204 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:35.192748 2578 scope.go:117] "RemoveContainer" containerID="3b3512a7f8a291923c56166df6d8ca7701103abf65f737aa03458d2965feb14d" Jul 22 16:14:35.221352 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:35.221329 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5664d4dcb6-6wxkb"] Jul 22 16:14:35.231871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:35.231850 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5664d4dcb6-6wxkb"] Jul 22 16:14:35.490396 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:35.490328 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27d0577-d0db-4d5a-ab75-61ed9cf86105" path="/var/lib/kubelet/pods/a27d0577-d0db-4d5a-ab75-61ed9cf86105/volumes" Jul 22 16:14:57.145512 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:57.145424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:57.299302 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:57.299265 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:14:58.266219 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:58.266187 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:14:58.266607 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:58.266447 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="alertmanager" containerID="cri-o://ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad" gracePeriod=120 Jul 22 16:14:58.266607 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:58.266542 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-metric" containerID="cri-o://3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d" gracePeriod=120 Jul 22 16:14:58.266607 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:58.266571 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="prom-label-proxy" containerID="cri-o://50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2" gracePeriod=120 Jul 22 16:14:58.266740 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:58.266622 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy" containerID="cri-o://b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c" gracePeriod=120 Jul 22 16:14:58.266740 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:58.266584 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-web" containerID="cri-o://df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e" gracePeriod=120 Jul 22 16:14:58.266740 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:58.266611 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="config-reloader" containerID="cri-o://9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281" gracePeriod=120 Jul 22 16:14:59.294677 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294640 2578 generic.go:356] "Generic (PLEG): container finished" podID="92baafac-4291-428f-a2d9-60f286971e35" containerID="50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2" exitCode=0 Jul 22 16:14:59.294677 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294667 2578 generic.go:356] "Generic (PLEG): container finished" podID="92baafac-4291-428f-a2d9-60f286971e35" containerID="b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c" exitCode=0 Jul 22 16:14:59.294677 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294674 2578 generic.go:356] "Generic (PLEG): container finished" podID="92baafac-4291-428f-a2d9-60f286971e35" containerID="9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281" exitCode=0 Jul 22 16:14:59.294677 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294682 2578 generic.go:356] "Generic (PLEG): container finished" podID="92baafac-4291-428f-a2d9-60f286971e35" containerID="ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad" exitCode=0 Jul 22 16:14:59.295124 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294712 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2"} Jul 22 16:14:59.295124 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c"} Jul 22 16:14:59.295124 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281"} Jul 22 16:14:59.295124 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.294887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad"} Jul 22 16:14:59.512128 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.512109 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:14:59.628881 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.628813 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-web\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.628881 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.628840 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-cluster-tls-config\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.628881 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.628863 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-config-volume\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.628905 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-metric\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.628965 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-metrics-client-ca\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.628994 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-tls-assets\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629021 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-web-config\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629050 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n27d7\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-kube-api-access-n27d7\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629081 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-trusted-ca-bundle\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629107 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-config-out\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629135 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-main-tls\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629546 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629185 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-main-db\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629546 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629220 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy\") pod \"92baafac-4291-428f-a2d9-60f286971e35\" (UID: \"92baafac-4291-428f-a2d9-60f286971e35\") " Jul 22 16:14:59.629653 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.629621 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:14:59.630097 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.630067 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:14:59.630618 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.630587 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:14:59.631485 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.631452 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:59.631730 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.631701 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:59.632000 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.631957 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:59.632000 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.631989 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-config-volume" (OuterVolumeSpecName: "config-volume") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:59.632115 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.632028 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:14:59.633016 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.632980 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-kube-api-access-n27d7" (OuterVolumeSpecName: "kube-api-access-n27d7") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "kube-api-access-n27d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:14:59.633016 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.632996 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-config-out" (OuterVolumeSpecName: "config-out") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:14:59.633580 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.633563 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:59.636303 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.636271 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:59.642674 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.642654 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-web-config" (OuterVolumeSpecName: "web-config") pod "92baafac-4291-428f-a2d9-60f286971e35" (UID: "92baafac-4291-428f-a2d9-60f286971e35"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:14:59.730741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730717 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730738 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730749 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-cluster-tls-config\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730758 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-config-volume\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730767 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730778 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-metrics-client-ca\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730786 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-tls-assets\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730794 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-web-config\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730802 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27d7\" (UniqueName: \"kubernetes.io/projected/92baafac-4291-428f-a2d9-60f286971e35-kube-api-access-n27d7\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730812 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730820 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-config-out\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730828 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92baafac-4291-428f-a2d9-60f286971e35-secret-alertmanager-main-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:14:59.730856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:14:59.730836 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92baafac-4291-428f-a2d9-60f286971e35-alertmanager-main-db\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:15:00.301255 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.301224 2578 generic.go:356] "Generic (PLEG): container finished" podID="92baafac-4291-428f-a2d9-60f286971e35" containerID="3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d" exitCode=0 Jul 22 16:15:00.301255 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.301249 2578 generic.go:356] "Generic (PLEG): container finished" podID="92baafac-4291-428f-a2d9-60f286971e35" containerID="df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e" exitCode=0 Jul 22 16:15:00.301695 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.301289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d"} Jul 22 16:15:00.301695 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.301295 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.301695 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.301341 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e"} Jul 22 16:15:00.301695 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.301367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92baafac-4291-428f-a2d9-60f286971e35","Type":"ContainerDied","Data":"8cda558e825c30eee0e64564566040b07ca0246e7215cf05dd1552159016f7fe"} Jul 22 16:15:00.301695 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.301397 2578 scope.go:117] "RemoveContainer" containerID="50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2" Jul 22 16:15:00.309047 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.309029 2578 scope.go:117] "RemoveContainer" containerID="3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d" Jul 22 16:15:00.315559 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.315513 2578 scope.go:117] "RemoveContainer" containerID="b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c" Jul 22 16:15:00.321878 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.321864 2578 scope.go:117] "RemoveContainer" containerID="df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e" Jul 22 16:15:00.332457 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.332436 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:15:00.334100 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.334081 2578 scope.go:117] "RemoveContainer" containerID="9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281" Jul 22 16:15:00.338294 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.338276 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:15:00.341084 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.341070 2578 scope.go:117] "RemoveContainer" containerID="ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad" Jul 22 16:15:00.347569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.347551 2578 scope.go:117] "RemoveContainer" containerID="68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf" Jul 22 16:15:00.353861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.353843 2578 scope.go:117] "RemoveContainer" containerID="50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2" Jul 22 16:15:00.354104 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:15:00.354087 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2\": container with ID starting with 50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2 not found: ID does not exist" containerID="50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2" Jul 22 16:15:00.354153 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354113 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2"} err="failed to get container status \"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2\": rpc error: code = NotFound desc = could not find container \"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2\": container with ID starting with 50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2 not found: ID does not exist" Jul 22 16:15:00.354153 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354129 2578 scope.go:117] "RemoveContainer" containerID="3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d" Jul 22 16:15:00.354331 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:15:00.354317 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d\": container with ID starting with 3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d not found: ID does not exist" containerID="3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d" Jul 22 16:15:00.354380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354333 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d"} err="failed to get container status \"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d\": rpc error: code = NotFound desc = could not find container \"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d\": container with ID starting with 3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d not found: ID does not exist" Jul 22 16:15:00.354380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354344 2578 scope.go:117] "RemoveContainer" containerID="b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c" Jul 22 16:15:00.354561 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:15:00.354546 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c\": container with ID starting with b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c not found: ID does not exist" containerID="b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c" Jul 22 16:15:00.354602 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354565 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c"} err="failed to get container status \"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c\": rpc error: code = NotFound desc = could not find container \"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c\": container with ID starting with b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c not found: ID does not exist" Jul 22 16:15:00.354602 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354578 2578 scope.go:117] "RemoveContainer" containerID="df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e" Jul 22 16:15:00.354762 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:15:00.354749 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e\": container with ID starting with df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e not found: ID does not exist" containerID="df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e" Jul 22 16:15:00.354799 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354764 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e"} err="failed to get container status \"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e\": rpc error: code = NotFound desc = could not find container \"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e\": container with ID starting with df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e not found: ID does not exist" Jul 22 16:15:00.354799 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354775 2578 scope.go:117] "RemoveContainer" containerID="9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281" Jul 22 16:15:00.354967 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:15:00.354949 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281\": container with ID starting with 9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281 not found: ID does not exist" containerID="9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281" Jul 22 16:15:00.355004 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354974 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281"} err="failed to get container status \"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281\": rpc error: code = NotFound desc = could not find container \"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281\": container with ID starting with 9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281 not found: ID does not exist" Jul 22 16:15:00.355004 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.354987 2578 scope.go:117] "RemoveContainer" containerID="ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad" Jul 22 16:15:00.355150 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:15:00.355137 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad\": container with ID starting with ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad not found: ID does not exist" containerID="ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad" Jul 22 16:15:00.355187 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355152 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad"} err="failed to get container status \"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad\": rpc error: code = NotFound desc = could not find container \"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad\": container with ID starting with ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad not found: ID does not exist" Jul 22 16:15:00.355187 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355163 2578 scope.go:117] "RemoveContainer" containerID="68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf" Jul 22 16:15:00.355329 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:15:00.355316 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf\": container with ID starting with 68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf not found: ID does not exist" containerID="68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf" Jul 22 16:15:00.355368 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355331 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf"} err="failed to get container status \"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf\": rpc error: code = NotFound desc = could not find container \"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf\": container with ID starting with 68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf not found: ID does not exist" Jul 22 16:15:00.355368 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355341 2578 scope.go:117] "RemoveContainer" containerID="50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2" Jul 22 16:15:00.355553 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355538 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2"} err="failed to get container status \"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2\": rpc error: code = NotFound desc = could not find container \"50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2\": container with ID starting with 50e689ad6a3619c5ed5951560c0c8c99459cd5a1b429a1697531e534607605c2 not found: ID does not exist" Jul 22 16:15:00.355596 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355553 2578 scope.go:117] "RemoveContainer" containerID="3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d" Jul 22 16:15:00.355759 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355739 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d"} err="failed to get container status \"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d\": rpc error: code = NotFound desc = could not find container \"3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d\": container with ID starting with 3882fa86c729d3924917eb1ab0bd144d096d29a9453fe6f54a0136ced22c5c1d not found: ID does not exist" Jul 22 16:15:00.355801 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355761 2578 scope.go:117] "RemoveContainer" containerID="b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c" Jul 22 16:15:00.355952 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355936 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c"} err="failed to get container status \"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c\": rpc error: code = NotFound desc = could not find container \"b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c\": container with ID starting with b7b09d612c52c2fe810b5feed2e5c1054f9fa7301f8489ce82825726eadf991c not found: ID does not exist" Jul 22 16:15:00.355991 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.355952 2578 scope.go:117] "RemoveContainer" containerID="df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e" Jul 22 16:15:00.356133 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.356113 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e"} err="failed to get container status \"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e\": rpc error: code = NotFound desc = could not find container \"df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e\": container with ID starting with df4777df6221e4995539fcea595c8b56e01943a14cffd5fd42a5c8dc3c47a05e not found: ID does not exist" Jul 22 16:15:00.356179 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.356134 2578 scope.go:117] "RemoveContainer" containerID="9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281" Jul 22 16:15:00.356347 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.356331 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281"} err="failed to get container status \"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281\": rpc error: code = NotFound desc = could not find container \"9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281\": container with ID starting with 9f20fd18efeab59e71b75d4a5b06dab5c16e4281257e8b11596291de285fb281 not found: ID does not exist" Jul 22 16:15:00.356347 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.356346 2578 scope.go:117] "RemoveContainer" containerID="ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad" Jul 22 16:15:00.356564 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.356541 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad"} err="failed to get container status \"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad\": rpc error: code = NotFound desc = could not find container \"ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad\": container with ID starting with ed3d72fefdf4ab8ba455507969f25565bddd304e08c9164028b03c14aef60bad not found: ID does not exist" Jul 22 16:15:00.356613 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.356565 2578 scope.go:117] "RemoveContainer" containerID="68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf" Jul 22 16:15:00.356774 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.356757 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf"} err="failed to get container status \"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf\": rpc error: code = NotFound desc = could not find container \"68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf\": container with ID starting with 68899680c333bcf9d9d9ad6eef8114be79d95c05316e2b7eed9daff8df716eaf not found: ID does not exist" Jul 22 16:15:00.395183 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395163 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:15:00.395421 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395408 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="alertmanager" Jul 22 16:15:00.395421 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395421 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="alertmanager" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395432 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-metric" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395439 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-metric" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395448 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="config-reloader" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395453 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="config-reloader" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395460 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="prom-label-proxy" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395465 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="prom-label-proxy" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395476 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="init-config-reloader" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395481 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="init-config-reloader" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395488 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2946fec9-cc76-45ff-ba02-077511935015" containerName="registry" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395493 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2946fec9-cc76-45ff-ba02-077511935015" containerName="registry" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395500 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27d0577-d0db-4d5a-ab75-61ed9cf86105" containerName="console" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395505 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27d0577-d0db-4d5a-ab75-61ed9cf86105" containerName="console" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395512 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-web" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395517 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-web" Jul 22 16:15:00.395514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395543 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395548 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395592 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="prom-label-proxy" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395600 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-metric" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395606 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="config-reloader" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395612 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2946fec9-cc76-45ff-ba02-077511935015" containerName="registry" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395618 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27d0577-d0db-4d5a-ab75-61ed9cf86105" containerName="console" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395624 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy-web" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395631 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="alertmanager" Jul 22 16:15:00.395967 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.395641 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92baafac-4291-428f-a2d9-60f286971e35" containerName="kube-rbac-proxy" Jul 22 16:15:00.398779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.398763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.414993 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.414977 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 16:15:00.414993 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.414987 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 16:15:00.415228 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.415214 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 16:15:00.415486 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.415471 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fkx9k\"" Jul 22 16:15:00.415584 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.415571 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 16:15:00.415717 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.415702 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 16:15:00.416211 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.416197 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 16:15:00.416289 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.416198 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 16:15:00.416289 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.416199 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 16:15:00.419807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.419789 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 16:15:00.433819 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.433792 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:15:00.434571 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aefbc532-4cd0-42d3-820c-70606655cff9-config-out\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434660 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434660 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aefbc532-4cd0-42d3-820c-70606655cff9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434660 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434781 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-web-config\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434781 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aefbc532-4cd0-42d3-820c-70606655cff9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434781 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-config-volume\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434781 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434941 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aefbc532-4cd0-42d3-820c-70606655cff9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434941 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434941 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434941 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aefbc532-4cd0-42d3-820c-70606655cff9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.434941 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.434933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29q4\" (UniqueName: \"kubernetes.io/projected/aefbc532-4cd0-42d3-820c-70606655cff9-kube-api-access-n29q4\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.535924 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.535901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aefbc532-4cd0-42d3-820c-70606655cff9-config-out\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536026 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.535930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536026 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.535958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aefbc532-4cd0-42d3-820c-70606655cff9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536026 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.535976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536026 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-web-config\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536234 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aefbc532-4cd0-42d3-820c-70606655cff9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536234 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-config-volume\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536234 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536234 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aefbc532-4cd0-42d3-820c-70606655cff9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536234 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536234 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536552 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aefbc532-4cd0-42d3-820c-70606655cff9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.536552 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.536280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29q4\" (UniqueName: \"kubernetes.io/projected/aefbc532-4cd0-42d3-820c-70606655cff9-kube-api-access-n29q4\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.537163 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.537140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aefbc532-4cd0-42d3-820c-70606655cff9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.537249 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.537225 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aefbc532-4cd0-42d3-820c-70606655cff9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.537249 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.537236 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aefbc532-4cd0-42d3-820c-70606655cff9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.538825 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.538798 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aefbc532-4cd0-42d3-820c-70606655cff9-config-out\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.538921 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.538856 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aefbc532-4cd0-42d3-820c-70606655cff9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.539285 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.539256 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.539768 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.539712 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-web-config\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.539768 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.539734 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-config-volume\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.539768 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.539756 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.539997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.539773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.540115 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.540091 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.540651 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.540628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aefbc532-4cd0-42d3-820c-70606655cff9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.547129 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.547108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29q4\" (UniqueName: \"kubernetes.io/projected/aefbc532-4cd0-42d3-820c-70606655cff9-kube-api-access-n29q4\") pod \"alertmanager-main-0\" (UID: \"aefbc532-4cd0-42d3-820c-70606655cff9\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.707266 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.707194 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:15:00.849976 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:00.849946 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:15:00.853974 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:15:00.853944 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefbc532_4cd0_42d3_820c_70606655cff9.slice/crio-98f3ffad605b59524eaae569808f8b9146f25adbb634bc69ba568d47e06eb8b4 WatchSource:0}: Error finding container 98f3ffad605b59524eaae569808f8b9146f25adbb634bc69ba568d47e06eb8b4: Status 404 returned error can't find the container with id 98f3ffad605b59524eaae569808f8b9146f25adbb634bc69ba568d47e06eb8b4 Jul 22 16:15:01.306324 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:01.306288 2578 generic.go:356] "Generic (PLEG): container finished" podID="aefbc532-4cd0-42d3-820c-70606655cff9" containerID="bcd081b974ea4b18353c7f36d646273bc9dd48b1454e6aea3567e481fa60a95a" exitCode=0 Jul 22 16:15:01.306791 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:01.306366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerDied","Data":"bcd081b974ea4b18353c7f36d646273bc9dd48b1454e6aea3567e481fa60a95a"} Jul 22 16:15:01.306791 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:01.306430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerStarted","Data":"98f3ffad605b59524eaae569808f8b9146f25adbb634bc69ba568d47e06eb8b4"} Jul 22 16:15:01.490742 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:01.490716 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92baafac-4291-428f-a2d9-60f286971e35" path="/var/lib/kubelet/pods/92baafac-4291-428f-a2d9-60f286971e35/volumes" Jul 22 16:15:02.314438 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.314403 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerStarted","Data":"d5bd9595e5ffe5247e7e98f8778f8bf6be14b003192a36ee394c0d6836d2f89b"} Jul 22 16:15:02.315038 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.314467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerStarted","Data":"dc3e20b0424c95a25e2a22626d45011f2b8485116d68c2bf7585bd4eaf3a2d6a"} Jul 22 16:15:02.315038 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.314492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerStarted","Data":"143486d1708f51df6a05fe5f60bc7936b251e4aced3865ccc1d8f694c571ec3f"} Jul 22 16:15:02.315038 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.314515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerStarted","Data":"ca83c24a8cd6ebbb8d9ad7a92ef08a1cd63284b5351b39ec991183ee52976099"} Jul 22 16:15:02.315038 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.314551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerStarted","Data":"2bd6b935e0b2d2b6bf00a72e95f4a287a39f40480c22725836e9fcffe5acd23e"} Jul 22 16:15:02.315038 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.314574 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aefbc532-4cd0-42d3-820c-70606655cff9","Type":"ContainerStarted","Data":"e9e37a89e56e81cf09b31bc95e3475e7411d3b7f579062e35d86c1e454859d77"} Jul 22 16:15:02.345185 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.345156 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6"] Jul 22 16:15:02.347598 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.347580 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.353750 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.353719 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 16:15:02.353861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.353723 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 16:15:02.353861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.353825 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 16:15:02.353861 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.353844 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tt5pp\"" Jul 22 16:15:02.354068 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.353827 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 16:15:02.354068 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.353882 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 16:15:02.364693 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.364675 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 16:15:02.373274 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.373253 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6"] Jul 22 16:15:02.398325 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.398282 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.384564556 podStartE2EDuration="2.398268861s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.704305ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:15:02.396462715 +0000 UTC m=+177.381346747" watchObservedRunningTime="2026-07-22 16:15:02.398268861 +0000 UTC m=+177.383152866" Jul 22 16:15:02.453122 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453088 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-metrics-client-ca\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.453248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-secret-telemeter-client\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.453248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.453248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-telemeter-client-tls\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.453248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.453248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-serving-certs-ca-bundle\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.453478 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt4j\" (UniqueName: \"kubernetes.io/projected/ec16a034-1158-4348-800f-915a01248b91-kube-api-access-jkt4j\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.453478 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.453352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-federate-client-tls\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-federate-client-tls\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554270 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-metrics-client-ca\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554270 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-secret-telemeter-client\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554270 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554404 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-telemeter-client-tls\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554404 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554404 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-serving-certs-ca-bundle\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.554404 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt4j\" (UniqueName: \"kubernetes.io/projected/ec16a034-1158-4348-800f-915a01248b91-kube-api-access-jkt4j\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.555188 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.554940 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-metrics-client-ca\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.555188 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.555132 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.555316 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.555202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec16a034-1158-4348-800f-915a01248b91-serving-certs-ca-bundle\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.557156 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.557128 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-telemeter-client-tls\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.557243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.557169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-secret-telemeter-client\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.557243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.557187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-federate-client-tls\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.557243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.557221 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec16a034-1158-4348-800f-915a01248b91-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.566146 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.566092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt4j\" (UniqueName: \"kubernetes.io/projected/ec16a034-1158-4348-800f-915a01248b91-kube-api-access-jkt4j\") pod \"telemeter-client-6d69d6bcdf-rqvb6\" (UID: \"ec16a034-1158-4348-800f-915a01248b91\") " pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.657700 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.657676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" Jul 22 16:15:02.787087 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:02.787004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6"] Jul 22 16:15:02.789070 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:15:02.789046 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec16a034_1158_4348_800f_915a01248b91.slice/crio-74345942e6950ffb63a626e93cdfa6a06e3fb461ef48b5fb110311bcfbf464a8 WatchSource:0}: Error finding container 74345942e6950ffb63a626e93cdfa6a06e3fb461ef48b5fb110311bcfbf464a8: Status 404 returned error can't find the container with id 74345942e6950ffb63a626e93cdfa6a06e3fb461ef48b5fb110311bcfbf464a8 Jul 22 16:15:03.319698 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:03.319661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" event={"ID":"ec16a034-1158-4348-800f-915a01248b91","Type":"ContainerStarted","Data":"74345942e6950ffb63a626e93cdfa6a06e3fb461ef48b5fb110311bcfbf464a8"} Jul 22 16:15:06.334883 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:06.334837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" event={"ID":"ec16a034-1158-4348-800f-915a01248b91","Type":"ContainerStarted","Data":"a5ef4e5402b1e56dfb769ed51764fa28ae145695414ea6382baa4b3b4635df4e"} Jul 22 16:15:06.335393 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:06.334904 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" event={"ID":"ec16a034-1158-4348-800f-915a01248b91","Type":"ContainerStarted","Data":"c8eda741a77e98b73136d5adc021ea3d3183782d740b8c07c1fb89439c85941d"} Jul 22 16:15:06.335393 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:06.334932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" event={"ID":"ec16a034-1158-4348-800f-915a01248b91","Type":"ContainerStarted","Data":"e1d119fca73c1064fe075e6832a802f7ad069c9924adc0a95efbd0e6b9dabe54"} Jul 22 16:15:06.362035 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:06.361981 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d69d6bcdf-rqvb6" podStartSLOduration=1.812598368 podStartE2EDuration="4.361965843s" totalImagesPullingTime="2.549367475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:15:06.361708891 +0000 UTC m=+181.346592914" watchObservedRunningTime="2026-07-22 16:15:06.361965843 +0000 UTC m=+181.346849849" Jul 22 16:15:07.399886 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.399858 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b6bbfcf96-x4d29"] Jul 22 16:15:07.402071 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.402055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.404874 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.404849 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 16:15:07.405385 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.405353 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 16:15:07.405488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.405357 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 16:15:07.406479 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.406459 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gb9f8\"" Jul 22 16:15:07.406574 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.406496 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 16:15:07.407736 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.407700 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 16:15:07.428506 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.428482 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 16:15:07.431803 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.431741 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6bbfcf96-x4d29"] Jul 22 16:15:07.493374 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.493347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-trusted-ca-bundle\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.493552 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.493391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjfxc\" (UniqueName: \"kubernetes.io/projected/7f297b43-daad-4cdd-bb76-1bfb71319f6e-kube-api-access-cjfxc\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.493552 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.493464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-oauth-serving-cert\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.493552 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.493503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-serving-cert\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.493730 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.493563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-config\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.493730 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.493655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-oauth-config\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.493730 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.493694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-service-ca\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.502468 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.502435 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:07.594316 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.594291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-trusted-ca-bundle\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.594410 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.594323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjfxc\" (UniqueName: \"kubernetes.io/projected/7f297b43-daad-4cdd-bb76-1bfb71319f6e-kube-api-access-cjfxc\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.594410 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.594346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-oauth-serving-cert\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.594410 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.594364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-serving-cert\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.594410 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.594384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-config\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.594642 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.594425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-oauth-config\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.594642 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.594455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-service-ca\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.595198 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.595169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-config\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.595277 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.595221 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-service-ca\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.595277 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.595226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-oauth-serving-cert\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.595357 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.595327 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-trusted-ca-bundle\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.596858 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.596840 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-oauth-config\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.597198 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.597182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-serving-cert\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.606057 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.606030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjfxc\" (UniqueName: \"kubernetes.io/projected/7f297b43-daad-4cdd-bb76-1bfb71319f6e-kube-api-access-cjfxc\") pod \"console-5b6bbfcf96-x4d29\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.711972 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.711918 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:07.842677 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.842656 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6bbfcf96-x4d29"] Jul 22 16:15:07.844758 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:15:07.844731 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f297b43_daad_4cdd_bb76_1bfb71319f6e.slice/crio-af8f0d540d4913b8bbbafede736063ee23cf650226352bf24661282e9bc026ab WatchSource:0}: Error finding container af8f0d540d4913b8bbbafede736063ee23cf650226352bf24661282e9bc026ab: Status 404 returned error can't find the container with id af8f0d540d4913b8bbbafede736063ee23cf650226352bf24661282e9bc026ab Jul 22 16:15:07.846804 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.846727 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:07.929880 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:07.929843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:08.189789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:08.189755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:08.343836 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:08.343807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bbfcf96-x4d29" event={"ID":"7f297b43-daad-4cdd-bb76-1bfb71319f6e","Type":"ContainerStarted","Data":"85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a"} Jul 22 16:15:08.343995 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:08.343852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bbfcf96-x4d29" event={"ID":"7f297b43-daad-4cdd-bb76-1bfb71319f6e","Type":"ContainerStarted","Data":"af8f0d540d4913b8bbbafede736063ee23cf650226352bf24661282e9bc026ab"} Jul 22 16:15:08.372246 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:08.372200 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b6bbfcf96-x4d29" podStartSLOduration=1.3721825619999999 podStartE2EDuration="1.372182562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:15:08.370258324 +0000 UTC m=+183.355142330" watchObservedRunningTime="2026-07-22 16:15:08.372182562 +0000 UTC m=+183.357066567" Jul 22 16:15:08.462306 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:08.462244 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:17.712701 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:17.712668 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:17.713099 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:17.712719 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:17.716876 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:17.716853 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:18.385469 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:18.385442 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:15:18.581362 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:18.581330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:18.684832 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:18.684750 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:18.782717 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:18.782683 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:36.686623 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:36.686583 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:45.833222 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:45.833180 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:15:45.935663 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:15:45.935628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:12.591609 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:12.591569 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:12.680702 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:12.680663 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:20.079173 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:20.079136 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:20.158938 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:20.158903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:24.422348 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:24.422314 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:24.498262 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:24.498222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:30.426762 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:30.426686 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6bbfcf96-x4d29"] Jul 22 16:16:30.540734 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:30.540703 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:30.645910 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:30.645871 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:30.802741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:30.802708 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:34.048751 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:34.048721 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:34.140566 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:34.140512 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:34.242340 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:34.242309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:16:55.446888 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.446827 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b6bbfcf96-x4d29" podUID="7f297b43-daad-4cdd-bb76-1bfb71319f6e" containerName="console" containerID="cri-o://85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a" gracePeriod=15 Jul 22 16:16:55.691427 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.691402 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:16:55.747964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.747908 2578 generic.go:356] "Generic (PLEG): container finished" podID="7f297b43-daad-4cdd-bb76-1bfb71319f6e" containerID="85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a" exitCode=0 Jul 22 16:16:55.747964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.747947 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bbfcf96-x4d29" event={"ID":"7f297b43-daad-4cdd-bb76-1bfb71319f6e","Type":"ContainerDied","Data":"85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a"} Jul 22 16:16:55.747964 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.747949 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bbfcf96-x4d29" Jul 22 16:16:55.748137 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.747979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bbfcf96-x4d29" event={"ID":"7f297b43-daad-4cdd-bb76-1bfb71319f6e","Type":"ContainerDied","Data":"af8f0d540d4913b8bbbafede736063ee23cf650226352bf24661282e9bc026ab"} Jul 22 16:16:55.748137 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.747998 2578 scope.go:117] "RemoveContainer" containerID="85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a" Jul 22 16:16:55.755587 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.755571 2578 scope.go:117] "RemoveContainer" containerID="85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a" Jul 22 16:16:55.755836 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:16:55.755816 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a\": container with ID starting with 85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a not found: ID does not exist" containerID="85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a" Jul 22 16:16:55.755894 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.755846 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a"} err="failed to get container status \"85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a\": rpc error: code = NotFound desc = could not find container \"85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a\": container with ID starting with 85b4ff8b8b9f09f7056f0ef081984d2a2f16ee8b1b25f9163b4c4fa761370a4a not found: ID does not exist" Jul 22 16:16:55.770119 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770099 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjfxc\" (UniqueName: \"kubernetes.io/projected/7f297b43-daad-4cdd-bb76-1bfb71319f6e-kube-api-access-cjfxc\") pod \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " Jul 22 16:16:55.770178 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770125 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-service-ca\") pod \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " Jul 22 16:16:55.770178 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770146 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-trusted-ca-bundle\") pod \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " Jul 22 16:16:55.770306 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770290 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-oauth-serving-cert\") pod \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " Jul 22 16:16:55.770348 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770318 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-config\") pod \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " Jul 22 16:16:55.770399 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770348 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-oauth-config\") pod \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " Jul 22 16:16:55.770452 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770408 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-serving-cert\") pod \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\" (UID: \"7f297b43-daad-4cdd-bb76-1bfb71319f6e\") " Jul 22 16:16:55.770507 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770451 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-service-ca" (OuterVolumeSpecName: "service-ca") pod "7f297b43-daad-4cdd-bb76-1bfb71319f6e" (UID: "7f297b43-daad-4cdd-bb76-1bfb71319f6e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:16:55.770703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770665 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7f297b43-daad-4cdd-bb76-1bfb71319f6e" (UID: "7f297b43-daad-4cdd-bb76-1bfb71319f6e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:16:55.770703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770688 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-service-ca\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:16:55.770840 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770781 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7f297b43-daad-4cdd-bb76-1bfb71319f6e" (UID: "7f297b43-daad-4cdd-bb76-1bfb71319f6e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:16:55.770840 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.770796 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-config" (OuterVolumeSpecName: "console-config") pod "7f297b43-daad-4cdd-bb76-1bfb71319f6e" (UID: "7f297b43-daad-4cdd-bb76-1bfb71319f6e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:16:55.772213 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.772192 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f297b43-daad-4cdd-bb76-1bfb71319f6e-kube-api-access-cjfxc" (OuterVolumeSpecName: "kube-api-access-cjfxc") pod "7f297b43-daad-4cdd-bb76-1bfb71319f6e" (UID: "7f297b43-daad-4cdd-bb76-1bfb71319f6e"). InnerVolumeSpecName "kube-api-access-cjfxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:16:55.772324 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.772301 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7f297b43-daad-4cdd-bb76-1bfb71319f6e" (UID: "7f297b43-daad-4cdd-bb76-1bfb71319f6e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:16:55.772445 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.772414 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7f297b43-daad-4cdd-bb76-1bfb71319f6e" (UID: "7f297b43-daad-4cdd-bb76-1bfb71319f6e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:16:55.871329 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.871301 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-oauth-serving-cert\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:16:55.871329 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.871324 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-config\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:16:55.871329 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.871335 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-oauth-config\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:16:55.871506 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.871344 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f297b43-daad-4cdd-bb76-1bfb71319f6e-console-serving-cert\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:16:55.871506 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.871353 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjfxc\" (UniqueName: \"kubernetes.io/projected/7f297b43-daad-4cdd-bb76-1bfb71319f6e-kube-api-access-cjfxc\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:16:55.871506 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:55.871361 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f297b43-daad-4cdd-bb76-1bfb71319f6e-trusted-ca-bundle\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:16:56.076571 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:56.076543 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6bbfcf96-x4d29"] Jul 22 16:16:56.081866 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:56.081845 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b6bbfcf96-x4d29"] Jul 22 16:16:57.490293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:16:57.490259 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f297b43-daad-4cdd-bb76-1bfb71319f6e" path="/var/lib/kubelet/pods/7f297b43-daad-4cdd-bb76-1bfb71319f6e/volumes" Jul 22 16:17:05.333858 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:17:05.333825 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:17:05.336594 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:17:05.336571 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:17:05.345403 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:17:05.345381 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:17:05.347744 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:17:05.347726 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:17:05.348120 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:17:05.348100 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 16:18:47.415741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:18:47.415704 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:18:48.946309 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:18:48.946272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:18:59.170911 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:18:59.170872 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:18:59.273168 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:18:59.273139 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:00.053736 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.053705 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:00.066708 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.066684 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28"] Jul 22 16:19:00.067034 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.067020 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f297b43-daad-4cdd-bb76-1bfb71319f6e" containerName="console" Jul 22 16:19:00.067080 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.067037 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f297b43-daad-4cdd-bb76-1bfb71319f6e" containerName="console" Jul 22 16:19:00.067115 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.067088 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f297b43-daad-4cdd-bb76-1bfb71319f6e" containerName="console" Jul 22 16:19:00.069857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.069843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:19:00.072606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.072586 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 22 16:19:00.072728 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.072612 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 22 16:19:00.074059 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.074043 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-26vql\"" Jul 22 16:19:00.095574 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.095549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28"] Jul 22 16:19:00.157785 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.157753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kqn\" (UniqueName: \"kubernetes.io/projected/ab4f4fb1-ebee-4504-adaa-b36b64f7dad0-kube-api-access-j8kqn\") pod \"feast-operator-controller-manager-65758586b6-bzq28\" (UID: \"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0\") " pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:19:00.258685 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.258655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kqn\" (UniqueName: \"kubernetes.io/projected/ab4f4fb1-ebee-4504-adaa-b36b64f7dad0-kube-api-access-j8kqn\") pod \"feast-operator-controller-manager-65758586b6-bzq28\" (UID: \"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0\") " pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:19:00.269704 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.269682 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kqn\" (UniqueName: \"kubernetes.io/projected/ab4f4fb1-ebee-4504-adaa-b36b64f7dad0-kube-api-access-j8kqn\") pod \"feast-operator-controller-manager-65758586b6-bzq28\" (UID: \"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0\") " pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:19:00.379068 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.379000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:19:00.513327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.513269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28"] Jul 22 16:19:00.515855 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:19:00.515832 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4f4fb1_ebee_4504_adaa_b36b64f7dad0.slice/crio-41bff6a89f84bfc0addf770ae285a906e5c33f429c2049eccb8ac90f911e23bb WatchSource:0}: Error finding container 41bff6a89f84bfc0addf770ae285a906e5c33f429c2049eccb8ac90f911e23bb: Status 404 returned error can't find the container with id 41bff6a89f84bfc0addf770ae285a906e5c33f429c2049eccb8ac90f911e23bb Jul 22 16:19:00.518165 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:00.518147 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:19:01.223101 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:01.223058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" event={"ID":"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0","Type":"ContainerStarted","Data":"41bff6a89f84bfc0addf770ae285a906e5c33f429c2049eccb8ac90f911e23bb"} Jul 22 16:19:02.559504 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:02.559465 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:02.662332 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:02.662302 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:03.234983 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:03.234897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" event={"ID":"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0","Type":"ContainerStarted","Data":"f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77"} Jul 22 16:19:03.234983 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:03.234964 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:19:03.256514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:03.256470 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" podStartSLOduration=0.812649623 podStartE2EDuration="3.256454077s" totalImagesPullingTime="2.443804454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:19:03.253555102 +0000 UTC m=+418.238439102" watchObservedRunningTime="2026-07-22 16:19:03.256454077 +0000 UTC m=+418.241338118" Jul 22 16:19:04.499259 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:04.499229 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:04.601404 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:04.601377 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:14.241857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:14.241827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:19:36.843749 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:36.843674 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:36.953747 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:36.953712 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:37.192946 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.192849 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:37.293272 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.293240 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:37.351550 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.351507 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7"] Jul 22 16:19:37.355404 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.355388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.358654 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.358632 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 22 16:19:37.358775 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.358718 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 22 16:19:37.358775 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.358723 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 22 16:19:37.358887 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.358723 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-7nnhd\"" Jul 22 16:19:37.358998 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.358980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 22 16:19:37.359054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.359002 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 22 16:19:37.359054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.359008 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 22 16:19:37.359134 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.358989 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 22 16:19:37.378935 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.378904 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7"] Jul 22 16:19:37.444554 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.444461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/21d35743-f12f-4d1c-9f94-8574251ec6d3-feast-data\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.444554 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.444505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.444741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.444554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49jd\" (UniqueName: \"kubernetes.io/projected/21d35743-f12f-4d1c-9f94-8574251ec6d3-kube-api-access-p49jd\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.444741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.444601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-online-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.444741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.444634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.444741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.444663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.540585 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.540554 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:37.545708 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.545689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.545774 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.545725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49jd\" (UniqueName: \"kubernetes.io/projected/21d35743-f12f-4d1c-9f94-8574251ec6d3-kube-api-access-p49jd\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.545774 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.545751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-online-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.545846 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:37.545835 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 22 16:19:37.545922 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:37.545914 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls podName:21d35743-f12f-4d1c-9f94-8574251ec6d3 nodeName:}" failed. No retries permitted until 2026-07-22 16:19:38.045893076 +0000 UTC m=+453.030777066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls") pod "feast-simple-feast-setup-8694747d5b-42rz7" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3") : secret "feast-simple-feast-setup-offline-tls" not found Jul 22 16:19:37.546027 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.545935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.546027 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.545980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.546129 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.546058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/21d35743-f12f-4d1c-9f94-8574251ec6d3-feast-data\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.546129 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:37.546114 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 22 16:19:37.546226 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:37.546126 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 22 16:19:37.546226 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:37.546171 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls podName:21d35743-f12f-4d1c-9f94-8574251ec6d3 nodeName:}" failed. No retries permitted until 2026-07-22 16:19:38.046154378 +0000 UTC m=+453.031038380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls") pod "feast-simple-feast-setup-8694747d5b-42rz7" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3") : secret "feast-simple-feast-setup-ui-tls" not found Jul 22 16:19:37.546226 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:37.546190 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls podName:21d35743-f12f-4d1c-9f94-8574251ec6d3 nodeName:}" failed. No retries permitted until 2026-07-22 16:19:38.046181333 +0000 UTC m=+453.031065319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls") pod "feast-simple-feast-setup-8694747d5b-42rz7" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3") : secret "feast-simple-feast-setup-registry-tls" not found Jul 22 16:19:37.546392 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.546376 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/21d35743-f12f-4d1c-9f94-8574251ec6d3-feast-data\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.548183 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.548158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-online-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.556175 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.556152 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49jd\" (UniqueName: \"kubernetes.io/projected/21d35743-f12f-4d1c-9f94-8574251ec6d3-kube-api-access-p49jd\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:37.664674 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.664639 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:37.766707 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:37.766667 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:38.050569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:38.050477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:38.050569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:38.050510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:38.050943 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:38.050576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:38.050943 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:38.050630 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 22 16:19:38.050943 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:19:38.050736 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls podName:21d35743-f12f-4d1c-9f94-8574251ec6d3 nodeName:}" failed. No retries permitted until 2026-07-22 16:19:39.050715419 +0000 UTC m=+454.035599403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls") pod "feast-simple-feast-setup-8694747d5b-42rz7" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3") : secret "feast-simple-feast-setup-ui-tls" not found Jul 22 16:19:38.052942 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:38.052911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:38.053049 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:38.052950 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:38.168441 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:38.168410 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:19:39.058583 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:39.058511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:39.060937 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:39.060900 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-42rz7\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:39.172767 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:39.172735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:19:39.313226 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:39.313195 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7"] Jul 22 16:19:39.314201 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:19:39.314177 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21d35743_f12f_4d1c_9f94_8574251ec6d3.slice/crio-137a74b1a99541540db8983f1e167193adc6c339e44a462334bf7ceed8dcd419 WatchSource:0}: Error finding container 137a74b1a99541540db8983f1e167193adc6c339e44a462334bf7ceed8dcd419: Status 404 returned error can't find the container with id 137a74b1a99541540db8983f1e167193adc6c339e44a462334bf7ceed8dcd419 Jul 22 16:19:39.376171 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:19:39.376134 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerStarted","Data":"137a74b1a99541540db8983f1e167193adc6c339e44a462334bf7ceed8dcd419"} Jul 22 16:20:07.509973 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:07.509936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerStarted","Data":"d5575a898887392ac8494682f9fa17441ab8ea9b4c288e1c65b501d10585c3e8"} Jul 22 16:20:12.529868 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:12.529830 2578 generic.go:356] "Generic (PLEG): container finished" podID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerID="d5575a898887392ac8494682f9fa17441ab8ea9b4c288e1c65b501d10585c3e8" exitCode=0 Jul 22 16:20:12.530272 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:12.529903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerDied","Data":"d5575a898887392ac8494682f9fa17441ab8ea9b4c288e1c65b501d10585c3e8"} Jul 22 16:20:13.535238 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:13.535205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerStarted","Data":"d10562709f969429c2006ac677bb15be0f4dfd75bd99bf427546c618ab323d4a"} Jul 22 16:20:16.380292 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:16.380256 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:20:16.467802 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:16.467767 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:20:18.558156 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:18.558124 2578 generic.go:356] "Generic (PLEG): container finished" podID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerID="d10562709f969429c2006ac677bb15be0f4dfd75bd99bf427546c618ab323d4a" exitCode=0 Jul 22 16:20:18.558508 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:18.558172 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerDied","Data":"d10562709f969429c2006ac677bb15be0f4dfd75bd99bf427546c618ab323d4a"} Jul 22 16:20:18.569985 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:18.569962 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 22 16:20:19.569095 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:19.569055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerStarted","Data":"0c7f9f49c5d5e5e4aedc232aea02810d5ad47ce3bbe17daad40df6421ceb1982"} Jul 22 16:20:19.569569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:19.569119 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerStarted","Data":"9b43bf3b9ac8355a06960afff9dcb6842f1be503a00955b9d3b93b99e66465de"} Jul 22 16:20:19.569569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:19.569139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerStarted","Data":"4b4729670168ba4db8dc3139efb223b079efcdd0d7b98ede6d5dd20de674a0a8"} Jul 22 16:20:19.569569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:19.569155 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerStarted","Data":"3918689cb44d08f953f5f96c4944ea89d430d91b0dedf4dd43725cd8931e6861"} Jul 22 16:20:19.609201 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:19.609128 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podStartSLOduration=14.616431299 podStartE2EDuration="42.609104963s" totalImagesPullingTime="27.992673664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:20:19.605738369 +0000 UTC m=+494.590622376" watchObservedRunningTime="2026-07-22 16:20:19.609104963 +0000 UTC m=+494.593988970" Jul 22 16:20:21.174543 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.173434 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:21.174543 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.173500 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:21.174543 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.173540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:21.174543 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.173560 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:21.175957 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.175924 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 22 16:20:21.176192 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.175979 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 22 16:20:21.176289 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.175976 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 22 16:20:21.176289 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:21.176017 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Jul 22 16:20:24.173121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:24.173064 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 22 16:20:24.173627 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:24.173135 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Jul 22 16:20:24.173627 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:24.173071 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 22 16:20:24.173627 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:24.173083 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 22 16:20:27.174355 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.174327 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.174721 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.174450 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.174721 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.174680 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.175082 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.174982 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.175082 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.175036 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.175082 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.175055 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.175307 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.175275 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.175402 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.175384 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.175454 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.175435 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.179379 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.179361 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.341932 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.341903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:20:27.443745 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.443650 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:20:27.543236 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.543202 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:20:27.608219 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.608192 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:27.611353 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:27.611329 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:20:28.176303 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:20:28.176268 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:30.630167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.630091 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:30.644424 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.644396 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7"] Jul 22 16:21:30.644674 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.644646 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="registry" containerID="cri-o://3918689cb44d08f953f5f96c4944ea89d430d91b0dedf4dd43725cd8931e6861" gracePeriod=30 Jul 22 16:21:30.644886 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.644708 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" containerID="cri-o://0c7f9f49c5d5e5e4aedc232aea02810d5ad47ce3bbe17daad40df6421ceb1982" gracePeriod=30 Jul 22 16:21:30.644991 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.644722 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="offline" containerID="cri-o://9b43bf3b9ac8355a06960afff9dcb6842f1be503a00955b9d3b93b99e66465de" gracePeriod=30 Jul 22 16:21:30.644991 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.644747 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" containerID="cri-o://4b4729670168ba4db8dc3139efb223b079efcdd0d7b98ede6d5dd20de674a0a8" gracePeriod=30 Jul 22 16:21:30.737255 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.737226 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:30.831592 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.831547 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:30.933956 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.933870 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:30.943243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.943214 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld"] Jul 22 16:21:30.947079 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.947062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:30.949912 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.949888 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-fb8j8\"" Jul 22 16:21:30.960062 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:30.960040 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld"] Jul 22 16:21:31.024054 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.024017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.024236 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.024076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.024236 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.024120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2febe58-9864-4c5e-8eb5-78f0b982bb43-feast-data\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.024236 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.024209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-registry-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.024375 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.024237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.024375 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.024277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvjk\" (UniqueName: \"kubernetes.io/projected/f2febe58-9864-4c5e-8eb5-78f0b982bb43-kube-api-access-9xvjk\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.046311 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.046280 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:31.125548 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.125484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.125741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.125576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.125741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.125602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2febe58-9864-4c5e-8eb5-78f0b982bb43-feast-data\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.125741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.125638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-registry-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.125741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.125661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.125741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.125699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvjk\" (UniqueName: \"kubernetes.io/projected/f2febe58-9864-4c5e-8eb5-78f0b982bb43-kube-api-access-9xvjk\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.125984 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:21:31.125738 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 22 16:21:31.125984 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:21:31.125790 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 22 16:21:31.125984 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:21:31.125838 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls podName:f2febe58-9864-4c5e-8eb5-78f0b982bb43 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:31.625815841 +0000 UTC m=+566.610699844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls") pod "feast-simple-feast-setup-8694747d5b-xrnld" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43") : secret "feast-simple-feast-setup-online-tls" not found Jul 22 16:21:31.125984 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:21:31.125877 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls podName:f2febe58-9864-4c5e-8eb5-78f0b982bb43 nodeName:}" failed. No retries permitted until 2026-07-22 16:21:31.625847808 +0000 UTC m=+566.610731794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls") pod "feast-simple-feast-setup-8694747d5b-xrnld" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43") : secret "feast-simple-feast-setup-offline-tls" not found Jul 22 16:21:31.126180 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.126040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2febe58-9864-4c5e-8eb5-78f0b982bb43-feast-data\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.128105 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.128077 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-registry-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.128239 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.128179 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-ui-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.135090 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.135063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvjk\" (UniqueName: \"kubernetes.io/projected/f2febe58-9864-4c5e-8eb5-78f0b982bb43-kube-api-access-9xvjk\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.151206 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.151174 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:31.250879 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.250852 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:31.307898 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.307860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:31.630465 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.630428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.630831 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.630484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.632725 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.632695 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.632822 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.632802 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls\") pod \"feast-simple-feast-setup-8694747d5b-xrnld\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.857113 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.857087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:31.865091 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.865069 2578 generic.go:356] "Generic (PLEG): container finished" podID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerID="0c7f9f49c5d5e5e4aedc232aea02810d5ad47ce3bbe17daad40df6421ceb1982" exitCode=0 Jul 22 16:21:31.865091 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.865088 2578 generic.go:356] "Generic (PLEG): container finished" podID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerID="4b4729670168ba4db8dc3139efb223b079efcdd0d7b98ede6d5dd20de674a0a8" exitCode=0 Jul 22 16:21:31.865281 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.865146 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerDied","Data":"0c7f9f49c5d5e5e4aedc232aea02810d5ad47ce3bbe17daad40df6421ceb1982"} Jul 22 16:21:31.865281 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.865178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerDied","Data":"4b4729670168ba4db8dc3139efb223b079efcdd0d7b98ede6d5dd20de674a0a8"} Jul 22 16:21:31.997213 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:31.997185 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld"] Jul 22 16:21:31.997869 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:21:31.997844 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2febe58_9864_4c5e_8eb5_78f0b982bb43.slice/crio-3c78708d888d9beca9efee753c805c70ca0802fa92f642fbf4b046f07bedac76 WatchSource:0}: Error finding container 3c78708d888d9beca9efee753c805c70ca0802fa92f642fbf4b046f07bedac76: Status 404 returned error can't find the container with id 3c78708d888d9beca9efee753c805c70ca0802fa92f642fbf4b046f07bedac76 Jul 22 16:21:32.870478 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:32.870430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerStarted","Data":"dab273b41f4f46831fa0d16b0f72b46d443e15eee5f10bebece7dcd0aa92f51a"} Jul 22 16:21:32.870478 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:32.870481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerStarted","Data":"3c78708d888d9beca9efee753c805c70ca0802fa92f642fbf4b046f07bedac76"} Jul 22 16:21:36.888619 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:36.888513 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerID="dab273b41f4f46831fa0d16b0f72b46d443e15eee5f10bebece7dcd0aa92f51a" exitCode=0 Jul 22 16:21:36.888619 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:36.888586 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerDied","Data":"dab273b41f4f46831fa0d16b0f72b46d443e15eee5f10bebece7dcd0aa92f51a"} Jul 22 16:21:37.179442 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:37.179352 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 22 16:21:37.608622 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:37.608567 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 22 16:21:37.894614 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:37.894513 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerStarted","Data":"e27e06acdbc180000b45afd003a6404ab518a990637a90303c02db5f22f27758"} Jul 22 16:21:42.917322 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:42.917287 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerID="e27e06acdbc180000b45afd003a6404ab518a990637a90303c02db5f22f27758" exitCode=0 Jul 22 16:21:42.917709 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:42.917360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerDied","Data":"e27e06acdbc180000b45afd003a6404ab518a990637a90303c02db5f22f27758"} Jul 22 16:21:43.928167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:43.928048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerStarted","Data":"991e04d97cf0d981a844a073cd0ee7f0c9ed45603d234d34e842d2f276cc342b"} Jul 22 16:21:43.928167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:43.928133 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerStarted","Data":"e1498839c486fd3cb67f1a52f7f45fb54a1c3e93891c536c705c8bea3f9178d0"} Jul 22 16:21:43.928167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:43.928155 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerStarted","Data":"1553ef55c9e660a7058bf3608395702e9c7b37948194862ff44a9bbe56e6eba2"} Jul 22 16:21:43.928776 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:43.928173 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerStarted","Data":"eac360b9bae4ced7338c1719e8715c1c2b05e727eb90a7b87096abd520165f94"} Jul 22 16:21:43.957444 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:43.957380 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podStartSLOduration=13.957359613 podStartE2EDuration="13.957359613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:21:43.955081865 +0000 UTC m=+578.939965872" watchObservedRunningTime="2026-07-22 16:21:43.957359613 +0000 UTC m=+578.942243621" Jul 22 16:21:46.858084 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.857436 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:46.858084 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.857496 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:46.858084 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.857514 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:46.858084 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.857546 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:46.859686 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.859644 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Jul 22 16:21:46.859937 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.859677 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Jul 22 16:21:46.860047 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.859696 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 22 16:21:46.860047 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:46.859752 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Jul 22 16:21:47.175876 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:47.175771 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 22 16:21:47.609035 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:47.608984 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 22 16:21:49.859012 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.858972 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.859457 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.859359 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.859457 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.859448 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.859761 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.859735 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.859898 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.859873 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.859952 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.859936 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.860031 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.859988 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.864247 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.864228 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.962935 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.962911 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.963084 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.962944 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.963217 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.963197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:49.965929 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:49.965912 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:21:50.305077 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:50.305047 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:21:57.175847 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:57.175804 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 22 16:21:57.176264 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:57.175949 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:21:57.608707 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:57.608663 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 22 16:21:57.608882 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:21:57.608798 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:22:01.013298 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.013264 2578 generic.go:356] "Generic (PLEG): container finished" podID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerID="9b43bf3b9ac8355a06960afff9dcb6842f1be503a00955b9d3b93b99e66465de" exitCode=137 Jul 22 16:22:01.013298 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.013291 2578 generic.go:356] "Generic (PLEG): container finished" podID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerID="3918689cb44d08f953f5f96c4944ea89d430d91b0dedf4dd43725cd8931e6861" exitCode=137 Jul 22 16:22:01.013682 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.013324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerDied","Data":"9b43bf3b9ac8355a06960afff9dcb6842f1be503a00955b9d3b93b99e66465de"} Jul 22 16:22:01.013682 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.013367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerDied","Data":"3918689cb44d08f953f5f96c4944ea89d430d91b0dedf4dd43725cd8931e6861"} Jul 22 16:22:01.293642 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.293620 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:22:01.372668 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.372630 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-online-tls\") pod \"21d35743-f12f-4d1c-9f94-8574251ec6d3\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " Jul 22 16:22:01.372668 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.372677 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls\") pod \"21d35743-f12f-4d1c-9f94-8574251ec6d3\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " Jul 22 16:22:01.372899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.372699 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls\") pod \"21d35743-f12f-4d1c-9f94-8574251ec6d3\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " Jul 22 16:22:01.372899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.372726 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p49jd\" (UniqueName: \"kubernetes.io/projected/21d35743-f12f-4d1c-9f94-8574251ec6d3-kube-api-access-p49jd\") pod \"21d35743-f12f-4d1c-9f94-8574251ec6d3\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " Jul 22 16:22:01.372899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.372790 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/21d35743-f12f-4d1c-9f94-8574251ec6d3-feast-data\") pod \"21d35743-f12f-4d1c-9f94-8574251ec6d3\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " Jul 22 16:22:01.372899 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.372832 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls\") pod \"21d35743-f12f-4d1c-9f94-8574251ec6d3\" (UID: \"21d35743-f12f-4d1c-9f94-8574251ec6d3\") " Jul 22 16:22:01.373417 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.373374 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d35743-f12f-4d1c-9f94-8574251ec6d3-feast-data" (OuterVolumeSpecName: "feast-data") pod "21d35743-f12f-4d1c-9f94-8574251ec6d3" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:22:01.374956 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.374929 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d35743-f12f-4d1c-9f94-8574251ec6d3-kube-api-access-p49jd" (OuterVolumeSpecName: "kube-api-access-p49jd") pod "21d35743-f12f-4d1c-9f94-8574251ec6d3" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3"). InnerVolumeSpecName "kube-api-access-p49jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:22:01.375212 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.375189 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "21d35743-f12f-4d1c-9f94-8574251ec6d3" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:22:01.375280 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.375217 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "21d35743-f12f-4d1c-9f94-8574251ec6d3" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:22:01.375280 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.375262 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-online-tls" (OuterVolumeSpecName: "online-tls") pod "21d35743-f12f-4d1c-9f94-8574251ec6d3" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:22:01.375393 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.375275 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "21d35743-f12f-4d1c-9f94-8574251ec6d3" (UID: "21d35743-f12f-4d1c-9f94-8574251ec6d3"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:22:01.473939 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.473914 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-online-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:22:01.473939 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.473939 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-registry-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:22:01.474064 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.473947 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-offline-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:22:01.474064 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.473956 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p49jd\" (UniqueName: \"kubernetes.io/projected/21d35743-f12f-4d1c-9f94-8574251ec6d3-kube-api-access-p49jd\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:22:01.474064 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.473966 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/21d35743-f12f-4d1c-9f94-8574251ec6d3-feast-data\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:22:01.474064 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:01.473974 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/21d35743-f12f-4d1c-9f94-8574251ec6d3-ui-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:22:02.022562 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.022506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" event={"ID":"21d35743-f12f-4d1c-9f94-8574251ec6d3","Type":"ContainerDied","Data":"137a74b1a99541540db8983f1e167193adc6c339e44a462334bf7ceed8dcd419"} Jul 22 16:22:02.022562 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.022549 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7" Jul 22 16:22:02.023100 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.022584 2578 scope.go:117] "RemoveContainer" containerID="0c7f9f49c5d5e5e4aedc232aea02810d5ad47ce3bbe17daad40df6421ceb1982" Jul 22 16:22:02.030877 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.030858 2578 scope.go:117] "RemoveContainer" containerID="9b43bf3b9ac8355a06960afff9dcb6842f1be503a00955b9d3b93b99e66465de" Jul 22 16:22:02.038032 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.038016 2578 scope.go:117] "RemoveContainer" containerID="4b4729670168ba4db8dc3139efb223b079efcdd0d7b98ede6d5dd20de674a0a8" Jul 22 16:22:02.044655 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.044636 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7"] Jul 22 16:22:02.045507 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.045490 2578 scope.go:117] "RemoveContainer" containerID="3918689cb44d08f953f5f96c4944ea89d430d91b0dedf4dd43725cd8931e6861" Jul 22 16:22:02.049968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.049951 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-42rz7"] Jul 22 16:22:02.053507 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.053489 2578 scope.go:117] "RemoveContainer" containerID="d10562709f969429c2006ac677bb15be0f4dfd75bd99bf427546c618ab323d4a" Jul 22 16:22:02.060180 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:02.060164 2578 scope.go:117] "RemoveContainer" containerID="d5575a898887392ac8494682f9fa17441ab8ea9b4c288e1c65b501d10585c3e8" Jul 22 16:22:03.490346 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:03.490312 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" path="/var/lib/kubelet/pods/21d35743-f12f-4d1c-9f94-8574251ec6d3/volumes" Jul 22 16:22:05.366918 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:05.366893 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:22:05.367319 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:05.366971 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:22:05.378336 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:05.378311 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:22:05.378451 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:05.378367 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:22:11.363112 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:11.363081 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:25.880878 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:25.880845 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:35.980455 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:35.980381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:52.501813 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.501780 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:52.602698 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.602662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:52.709621 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.709586 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:52.814751 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.814717 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:52.831216 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831188 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz"] Jul 22 16:22:52.831558 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831545 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" Jul 22 16:22:52.831615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831559 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" Jul 22 16:22:52.831615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831568 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" Jul 22 16:22:52.831615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831574 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" Jul 22 16:22:52.831615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831585 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="feast-init" Jul 22 16:22:52.831615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831591 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="feast-init" Jul 22 16:22:52.831615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831597 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="offline" Jul 22 16:22:52.831615 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831602 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="offline" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831619 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="registry" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831624 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="registry" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831632 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="feast-apply" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831638 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="feast-apply" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831691 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="registry" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831702 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="online" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831712 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="offline" Jul 22 16:22:52.831843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.831721 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d35743-f12f-4d1c-9f94-8574251ec6d3" containerName="ui" Jul 22 16:22:52.834962 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.834946 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:52.837367 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.837346 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 22 16:22:52.837367 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.837366 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 22 16:22:52.837513 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.837417 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 22 16:22:52.837513 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.837494 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 22 16:22:52.837811 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.837796 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 22 16:22:52.837857 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.837802 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 22 16:22:52.838380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.838361 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-9jl8h\"" Jul 22 16:22:52.838380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.838369 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 22 16:22:52.853274 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.853253 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz"] Jul 22 16:22:52.884445 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.884418 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:52.935606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.935579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzz5\" (UniqueName: \"kubernetes.io/projected/46ef6c05-c689-4243-9096-84b51355c395-kube-api-access-xxzz5\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:52.935702 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.935637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-offline-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:52.935702 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.935670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:52.935786 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.935728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ef6c05-c689-4243-9096-84b51355c395-feast-data\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:52.935786 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.935758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-online-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:52.935865 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.935790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/46ef6c05-c689-4243-9096-84b51355c395-registry-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:52.975111 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:52.975090 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:53.037004 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.036977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-offline-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.037118 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.037008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.037118 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.037029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ef6c05-c689-4243-9096-84b51355c395-feast-data\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.037118 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.037057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-online-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.037118 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.037100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/46ef6c05-c689-4243-9096-84b51355c395-registry-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.037354 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:22:53.037132 2578 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 22 16:22:53.037354 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.037139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzz5\" (UniqueName: \"kubernetes.io/projected/46ef6c05-c689-4243-9096-84b51355c395-kube-api-access-xxzz5\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.037354 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:22:53.037201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls podName:46ef6c05-c689-4243-9096-84b51355c395 nodeName:}" failed. No retries permitted until 2026-07-22 16:22:53.5371821 +0000 UTC m=+648.522066099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls") pod "feast-simple-feast-remote-setup-85967c56d5-gk6pz" (UID: "46ef6c05-c689-4243-9096-84b51355c395") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 22 16:22:53.037583 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.037400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ef6c05-c689-4243-9096-84b51355c395-feast-data\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.037792 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.037773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/46ef6c05-c689-4243-9096-84b51355c395-registry-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.039406 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.039384 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-offline-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.039504 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.039487 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-online-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.046245 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.046223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzz5\" (UniqueName: \"kubernetes.io/projected/46ef6c05-c689-4243-9096-84b51355c395-kube-api-access-xxzz5\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.120551 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.120454 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:22:53.540400 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.540371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.542756 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.542738 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls\") pod \"feast-simple-feast-remote-setup-85967c56d5-gk6pz\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.744138 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.744101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:22:53.882334 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:53.882306 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz"] Jul 22 16:22:53.883776 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:22:53.883754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46ef6c05_c689_4243_9096_84b51355c395.slice/crio-0ce7ba571040e5fe4e59a81a9398cc0e741d5a514b35b84256f01b85cf93ff43 WatchSource:0}: Error finding container 0ce7ba571040e5fe4e59a81a9398cc0e741d5a514b35b84256f01b85cf93ff43: Status 404 returned error can't find the container with id 0ce7ba571040e5fe4e59a81a9398cc0e741d5a514b35b84256f01b85cf93ff43 Jul 22 16:22:54.233073 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:54.232987 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerStarted","Data":"98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a"} Jul 22 16:22:54.233073 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:54.233043 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerStarted","Data":"0ce7ba571040e5fe4e59a81a9398cc0e741d5a514b35b84256f01b85cf93ff43"} Jul 22 16:22:59.254950 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:59.254914 2578 generic.go:356] "Generic (PLEG): container finished" podID="46ef6c05-c689-4243-9096-84b51355c395" containerID="98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a" exitCode=0 Jul 22 16:22:59.255453 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:22:59.254966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerDied","Data":"98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a"} Jul 22 16:23:00.261507 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:00.261471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerStarted","Data":"66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3"} Jul 22 16:23:05.287588 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:05.287552 2578 generic.go:356] "Generic (PLEG): container finished" podID="46ef6c05-c689-4243-9096-84b51355c395" containerID="66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3" exitCode=0 Jul 22 16:23:05.288032 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:05.287607 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerDied","Data":"66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3"} Jul 22 16:23:06.295221 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:06.295186 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerStarted","Data":"c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180"} Jul 22 16:23:06.295730 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:06.295244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerStarted","Data":"886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62"} Jul 22 16:23:06.295730 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:06.295260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerStarted","Data":"86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3"} Jul 22 16:23:06.323196 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:06.323136 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podStartSLOduration=14.323115198 podStartE2EDuration="14.323115198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:23:06.319722744 +0000 UTC m=+661.304606749" watchObservedRunningTime="2026-07-22 16:23:06.323115198 +0000 UTC m=+661.307999205" Jul 22 16:23:08.744833 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:08.744790 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:08.745332 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:08.744944 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:08.745332 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:08.744970 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:08.746991 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:08.746950 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Jul 22 16:23:08.747130 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:08.746990 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 22 16:23:08.747130 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:08.747015 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 22 16:23:11.745780 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:11.745751 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:11.746243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:11.745871 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:11.746243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:11.746219 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:11.746357 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:11.746249 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:11.746449 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:11.746433 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:11.746670 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:11.746649 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:11.752050 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:11.752031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:12.329944 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:12.329913 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:12.334394 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:12.334370 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:23:12.617971 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:23:12.617899 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:24:14.092355 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.092270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:24:14.100568 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.100537 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz"] Jul 22 16:24:14.100815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.100762 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" containerID="cri-o://86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3" gracePeriod=30 Jul 22 16:24:14.101028 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.100828 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" containerID="cri-o://c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180" gracePeriod=30 Jul 22 16:24:14.101109 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.100855 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="offline" containerID="cri-o://886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62" gracePeriod=30 Jul 22 16:24:14.208060 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.208025 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld"] Jul 22 16:24:14.208496 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.208297 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="registry" containerID="cri-o://eac360b9bae4ced7338c1719e8715c1c2b05e727eb90a7b87096abd520165f94" gracePeriod=30 Jul 22 16:24:14.208496 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.208340 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" containerID="cri-o://1553ef55c9e660a7058bf3608395702e9c7b37948194862ff44a9bbe56e6eba2" gracePeriod=30 Jul 22 16:24:14.208496 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.208355 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="offline" containerID="cri-o://e1498839c486fd3cb67f1a52f7f45fb54a1c3e93891c536c705c8bea3f9178d0" gracePeriod=30 Jul 22 16:24:14.208496 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:14.208364 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" containerID="cri-o://991e04d97cf0d981a844a073cd0ee7f0c9ed45603d234d34e842d2f276cc342b" gracePeriod=30 Jul 22 16:24:15.408662 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.408621 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:24:15.588143 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.588111 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerID="991e04d97cf0d981a844a073cd0ee7f0c9ed45603d234d34e842d2f276cc342b" exitCode=0 Jul 22 16:24:15.588143 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.588135 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerID="1553ef55c9e660a7058bf3608395702e9c7b37948194862ff44a9bbe56e6eba2" exitCode=0 Jul 22 16:24:15.588350 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.588186 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerDied","Data":"991e04d97cf0d981a844a073cd0ee7f0c9ed45603d234d34e842d2f276cc342b"} Jul 22 16:24:15.588350 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.588228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerDied","Data":"1553ef55c9e660a7058bf3608395702e9c7b37948194862ff44a9bbe56e6eba2"} Jul 22 16:24:15.590702 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.590671 2578 generic.go:356] "Generic (PLEG): container finished" podID="46ef6c05-c689-4243-9096-84b51355c395" containerID="c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180" exitCode=0 Jul 22 16:24:15.590702 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.590692 2578 generic.go:356] "Generic (PLEG): container finished" podID="46ef6c05-c689-4243-9096-84b51355c395" containerID="86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3" exitCode=0 Jul 22 16:24:15.590856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.590716 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerDied","Data":"c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180"} Jul 22 16:24:15.590856 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:15.590758 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerDied","Data":"86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3"} Jul 22 16:24:19.383499 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:19.383466 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:24:19.859588 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:19.859546 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Jul 22 16:24:19.963981 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:19.963936 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 22 16:24:21.746840 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:21.746784 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 22 16:24:22.330598 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:22.330551 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 22 16:24:29.860088 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:29.860036 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Jul 22 16:24:29.963396 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:29.963350 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 22 16:24:31.747116 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:31.747070 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 22 16:24:32.330850 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:32.330806 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 22 16:24:39.860489 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:39.860443 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused" Jul 22 16:24:39.860972 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:39.860625 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:24:39.963244 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:39.963196 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Jul 22 16:24:39.963430 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:39.963327 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:24:41.747161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:41.747120 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 22 16:24:41.747634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:41.747252 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:24:42.330159 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:42.330117 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 22 16:24:42.330336 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:42.330269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:24:44.323504 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.323480 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:24:44.457121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457030 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ef6c05-c689-4243-9096-84b51355c395-feast-data\") pod \"46ef6c05-c689-4243-9096-84b51355c395\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " Jul 22 16:24:44.457121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457072 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls\") pod \"46ef6c05-c689-4243-9096-84b51355c395\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " Jul 22 16:24:44.457121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457100 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/46ef6c05-c689-4243-9096-84b51355c395-registry-tls\") pod \"46ef6c05-c689-4243-9096-84b51355c395\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " Jul 22 16:24:44.457403 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxzz5\" (UniqueName: \"kubernetes.io/projected/46ef6c05-c689-4243-9096-84b51355c395-kube-api-access-xxzz5\") pod \"46ef6c05-c689-4243-9096-84b51355c395\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " Jul 22 16:24:44.457403 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457158 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-offline-tls\") pod \"46ef6c05-c689-4243-9096-84b51355c395\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " Jul 22 16:24:44.457403 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457188 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-online-tls\") pod \"46ef6c05-c689-4243-9096-84b51355c395\" (UID: \"46ef6c05-c689-4243-9096-84b51355c395\") " Jul 22 16:24:44.457594 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457552 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ef6c05-c689-4243-9096-84b51355c395-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "46ef6c05-c689-4243-9096-84b51355c395" (UID: "46ef6c05-c689-4243-9096-84b51355c395"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:24:44.457672 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.457642 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ef6c05-c689-4243-9096-84b51355c395-feast-data" (OuterVolumeSpecName: "feast-data") pod "46ef6c05-c689-4243-9096-84b51355c395" (UID: "46ef6c05-c689-4243-9096-84b51355c395"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:24:44.459372 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.459336 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "46ef6c05-c689-4243-9096-84b51355c395" (UID: "46ef6c05-c689-4243-9096-84b51355c395"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:44.459372 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.459356 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "46ef6c05-c689-4243-9096-84b51355c395" (UID: "46ef6c05-c689-4243-9096-84b51355c395"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:44.459568 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.459545 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-online-tls" (OuterVolumeSpecName: "online-tls") pod "46ef6c05-c689-4243-9096-84b51355c395" (UID: "46ef6c05-c689-4243-9096-84b51355c395"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:44.459611 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.459572 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ef6c05-c689-4243-9096-84b51355c395-kube-api-access-xxzz5" (OuterVolumeSpecName: "kube-api-access-xxzz5") pod "46ef6c05-c689-4243-9096-84b51355c395" (UID: "46ef6c05-c689-4243-9096-84b51355c395"). InnerVolumeSpecName "kube-api-access-xxzz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:24:44.557929 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.557897 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/46ef6c05-c689-4243-9096-84b51355c395-feast-data\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:44.557929 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.557924 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-ui-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:44.557929 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.557932 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/46ef6c05-c689-4243-9096-84b51355c395-registry-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:44.558155 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.557942 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzz5\" (UniqueName: \"kubernetes.io/projected/46ef6c05-c689-4243-9096-84b51355c395-kube-api-access-xxzz5\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:44.558155 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.557950 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-offline-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:44.558155 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.557958 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/46ef6c05-c689-4243-9096-84b51355c395-online-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:44.708968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.708881 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerID="e1498839c486fd3cb67f1a52f7f45fb54a1c3e93891c536c705c8bea3f9178d0" exitCode=137 Jul 22 16:24:44.708968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.708907 2578 generic.go:356] "Generic (PLEG): container finished" podID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerID="eac360b9bae4ced7338c1719e8715c1c2b05e727eb90a7b87096abd520165f94" exitCode=137 Jul 22 16:24:44.708968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.708955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerDied","Data":"e1498839c486fd3cb67f1a52f7f45fb54a1c3e93891c536c705c8bea3f9178d0"} Jul 22 16:24:44.709212 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.708999 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerDied","Data":"eac360b9bae4ced7338c1719e8715c1c2b05e727eb90a7b87096abd520165f94"} Jul 22 16:24:44.711235 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.711210 2578 generic.go:356] "Generic (PLEG): container finished" podID="46ef6c05-c689-4243-9096-84b51355c395" containerID="886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62" exitCode=137 Jul 22 16:24:44.711353 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.711279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerDied","Data":"886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62"} Jul 22 16:24:44.711353 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.711284 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" Jul 22 16:24:44.711353 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.711310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz" event={"ID":"46ef6c05-c689-4243-9096-84b51355c395","Type":"ContainerDied","Data":"0ce7ba571040e5fe4e59a81a9398cc0e741d5a514b35b84256f01b85cf93ff43"} Jul 22 16:24:44.711353 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.711330 2578 scope.go:117] "RemoveContainer" containerID="c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180" Jul 22 16:24:44.720692 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.720674 2578 scope.go:117] "RemoveContainer" containerID="886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62" Jul 22 16:24:44.730813 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.730790 2578 scope.go:117] "RemoveContainer" containerID="86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3" Jul 22 16:24:44.740422 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.740387 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz"] Jul 22 16:24:44.740486 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.740455 2578 scope.go:117] "RemoveContainer" containerID="66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3" Jul 22 16:24:44.742048 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.742022 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-85967c56d5-gk6pz"] Jul 22 16:24:44.748492 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.748476 2578 scope.go:117] "RemoveContainer" containerID="98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a" Jul 22 16:24:44.756127 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.756103 2578 scope.go:117] "RemoveContainer" containerID="c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180" Jul 22 16:24:44.756414 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:24:44.756396 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180\": container with ID starting with c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180 not found: ID does not exist" containerID="c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180" Jul 22 16:24:44.756487 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.756425 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180"} err="failed to get container status \"c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180\": rpc error: code = NotFound desc = could not find container \"c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180\": container with ID starting with c168a1305c90a6452493efe18968aa3ca4c9ef915e6bd5babdcca8a9fc0ad180 not found: ID does not exist" Jul 22 16:24:44.756487 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.756446 2578 scope.go:117] "RemoveContainer" containerID="886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62" Jul 22 16:24:44.756727 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:24:44.756709 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62\": container with ID starting with 886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62 not found: ID does not exist" containerID="886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62" Jul 22 16:24:44.756792 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.756736 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62"} err="failed to get container status \"886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62\": rpc error: code = NotFound desc = could not find container \"886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62\": container with ID starting with 886275d51d77c0deb0a3b798d4ab686e4ca2f3246b992b527775bc9f04089f62 not found: ID does not exist" Jul 22 16:24:44.756792 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.756759 2578 scope.go:117] "RemoveContainer" containerID="86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3" Jul 22 16:24:44.756990 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:24:44.756972 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3\": container with ID starting with 86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3 not found: ID does not exist" containerID="86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3" Jul 22 16:24:44.757055 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.756999 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3"} err="failed to get container status \"86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3\": rpc error: code = NotFound desc = could not find container \"86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3\": container with ID starting with 86e85ae44e8b4dcf34f04d7f6137fd03110a616dc80107a57b49b5161c2987c3 not found: ID does not exist" Jul 22 16:24:44.757055 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.757018 2578 scope.go:117] "RemoveContainer" containerID="66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3" Jul 22 16:24:44.757252 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:24:44.757235 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3\": container with ID starting with 66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3 not found: ID does not exist" containerID="66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3" Jul 22 16:24:44.757297 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.757257 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3"} err="failed to get container status \"66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3\": rpc error: code = NotFound desc = could not find container \"66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3\": container with ID starting with 66f36ae991251fc9810e46eba1b97fd890119a38f3ba7238a1baf078638fbce3 not found: ID does not exist" Jul 22 16:24:44.757297 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.757273 2578 scope.go:117] "RemoveContainer" containerID="98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a" Jul 22 16:24:44.757562 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:24:44.757537 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a\": container with ID starting with 98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a not found: ID does not exist" containerID="98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a" Jul 22 16:24:44.757668 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.757565 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a"} err="failed to get container status \"98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a\": rpc error: code = NotFound desc = could not find container \"98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a\": container with ID starting with 98cd82e5e6b7a5535e1b9f1a3d66aa4cc267f5606b7253dc7565347420a46c4a not found: ID does not exist" Jul 22 16:24:44.865343 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.865318 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:24:44.959944 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.959863 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xvjk\" (UniqueName: \"kubernetes.io/projected/f2febe58-9864-4c5e-8eb5-78f0b982bb43-kube-api-access-9xvjk\") pod \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " Jul 22 16:24:44.959944 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.959922 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls\") pod \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " Jul 22 16:24:44.960154 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.959961 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls\") pod \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " Jul 22 16:24:44.960154 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.960034 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2febe58-9864-4c5e-8eb5-78f0b982bb43-feast-data\") pod \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " Jul 22 16:24:44.960154 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.960062 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-ui-tls\") pod \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " Jul 22 16:24:44.960154 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.960094 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-registry-tls\") pod \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\" (UID: \"f2febe58-9864-4c5e-8eb5-78f0b982bb43\") " Jul 22 16:24:44.960657 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.960626 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2febe58-9864-4c5e-8eb5-78f0b982bb43-feast-data" (OuterVolumeSpecName: "feast-data") pod "f2febe58-9864-4c5e-8eb5-78f0b982bb43" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:24:44.962002 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.961973 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f2febe58-9864-4c5e-8eb5-78f0b982bb43" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:44.962500 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.962475 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls" (OuterVolumeSpecName: "online-tls") pod "f2febe58-9864-4c5e-8eb5-78f0b982bb43" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:44.962500 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.962488 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f2febe58-9864-4c5e-8eb5-78f0b982bb43" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:44.962656 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.962597 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f2febe58-9864-4c5e-8eb5-78f0b982bb43" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:24:44.962694 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:44.962663 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2febe58-9864-4c5e-8eb5-78f0b982bb43-kube-api-access-9xvjk" (OuterVolumeSpecName: "kube-api-access-9xvjk") pod "f2febe58-9864-4c5e-8eb5-78f0b982bb43" (UID: "f2febe58-9864-4c5e-8eb5-78f0b982bb43"). InnerVolumeSpecName "kube-api-access-9xvjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:24:45.060703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.060672 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-online-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:45.060703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.060699 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f2febe58-9864-4c5e-8eb5-78f0b982bb43-feast-data\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:45.060703 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.060707 2578 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-ui-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:45.060923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.060716 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-registry-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:45.060923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.060724 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xvjk\" (UniqueName: \"kubernetes.io/projected/f2febe58-9864-4c5e-8eb5-78f0b982bb43-kube-api-access-9xvjk\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:45.060923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.060732 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f2febe58-9864-4c5e-8eb5-78f0b982bb43-offline-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:24:45.490466 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.490434 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ef6c05-c689-4243-9096-84b51355c395" path="/var/lib/kubelet/pods/46ef6c05-c689-4243-9096-84b51355c395/volumes" Jul 22 16:24:45.723142 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.723106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" event={"ID":"f2febe58-9864-4c5e-8eb5-78f0b982bb43","Type":"ContainerDied","Data":"3c78708d888d9beca9efee753c805c70ca0802fa92f642fbf4b046f07bedac76"} Jul 22 16:24:45.723327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.723158 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld" Jul 22 16:24:45.723327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.723170 2578 scope.go:117] "RemoveContainer" containerID="991e04d97cf0d981a844a073cd0ee7f0c9ed45603d234d34e842d2f276cc342b" Jul 22 16:24:45.731457 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.731437 2578 scope.go:117] "RemoveContainer" containerID="e1498839c486fd3cb67f1a52f7f45fb54a1c3e93891c536c705c8bea3f9178d0" Jul 22 16:24:45.739105 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.739090 2578 scope.go:117] "RemoveContainer" containerID="1553ef55c9e660a7058bf3608395702e9c7b37948194862ff44a9bbe56e6eba2" Jul 22 16:24:45.745928 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.745913 2578 scope.go:117] "RemoveContainer" containerID="eac360b9bae4ced7338c1719e8715c1c2b05e727eb90a7b87096abd520165f94" Jul 22 16:24:45.749696 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.749676 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld"] Jul 22 16:24:45.753982 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.753596 2578 scope.go:117] "RemoveContainer" containerID="e27e06acdbc180000b45afd003a6404ab518a990637a90303c02db5f22f27758" Jul 22 16:24:45.756439 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.756420 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8694747d5b-xrnld"] Jul 22 16:24:45.760916 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:45.760899 2578 scope.go:117] "RemoveContainer" containerID="dab273b41f4f46831fa0d16b0f72b46d443e15eee5f10bebece7dcd0aa92f51a" Jul 22 16:24:47.490449 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:47.490417 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" path="/var/lib/kubelet/pods/f2febe58-9864-4c5e-8eb5-78f0b982bb43/volumes" Jul 22 16:24:57.329057 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.329021 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:24:57.571022 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.570996 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-jv9cp"] Jul 22 16:24:57.571293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571281 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="feast-init" Jul 22 16:24:57.571341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571295 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="feast-init" Jul 22 16:24:57.571341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571306 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="feast-apply" Jul 22 16:24:57.571341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571312 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="feast-apply" Jul 22 16:24:57.571341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571322 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" Jul 22 16:24:57.571341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571327 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" Jul 22 16:24:57.571341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571334 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="offline" Jul 22 16:24:57.571341 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571339 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="offline" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571346 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571351 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571359 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="feast-init" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571365 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="feast-init" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571378 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="offline" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571383 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="offline" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571389 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571394 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571400 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571404 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571413 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="feast-apply" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571419 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="feast-apply" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571425 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="registry" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571429 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="registry" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571488 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="online" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571497 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="registry" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571502 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="ui" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571508 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="online" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571512 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="offline" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571538 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ef6c05-c689-4243-9096-84b51355c395" containerName="offline" Jul 22 16:24:57.571634 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.571549 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2febe58-9864-4c5e-8eb5-78f0b982bb43" containerName="ui" Jul 22 16:24:57.576206 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.576189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:57.578824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.578800 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 22 16:24:57.578954 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.578942 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 22 16:24:57.579114 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.579070 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 22 16:24:57.580153 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.580132 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-ssglr\"" Jul 22 16:24:57.582330 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.582308 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jv9cp"] Jul 22 16:24:57.610544 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.610502 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:24:57.662200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.662171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8c3ca48-7d66-4816-81a2-d907c407df57-postgresdata\") pod \"postgres-864846d4f5-jv9cp\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:57.662331 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.662226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9x6\" (UniqueName: \"kubernetes.io/projected/c8c3ca48-7d66-4816-81a2-d907c407df57-kube-api-access-fg9x6\") pod \"postgres-864846d4f5-jv9cp\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:57.671243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.671215 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:24:57.762904 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.762877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8c3ca48-7d66-4816-81a2-d907c407df57-postgresdata\") pod \"postgres-864846d4f5-jv9cp\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:57.763030 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.762944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9x6\" (UniqueName: \"kubernetes.io/projected/c8c3ca48-7d66-4816-81a2-d907c407df57-kube-api-access-fg9x6\") pod \"postgres-864846d4f5-jv9cp\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:57.763228 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.763208 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8c3ca48-7d66-4816-81a2-d907c407df57-postgresdata\") pod \"postgres-864846d4f5-jv9cp\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:57.772492 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.772465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9x6\" (UniqueName: \"kubernetes.io/projected/c8c3ca48-7d66-4816-81a2-d907c407df57-kube-api-access-fg9x6\") pod \"postgres-864846d4f5-jv9cp\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:57.894643 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:57.894576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:24:58.018348 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:58.018319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jv9cp"] Jul 22 16:24:58.021183 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:24:58.021154 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8c3ca48_7d66_4816_81a2_d907c407df57.slice/crio-082cbfe190b98b5a049244a3b4d2b28ab47ed4f26674eac44546e96a8d4110e5 WatchSource:0}: Error finding container 082cbfe190b98b5a049244a3b4d2b28ab47ed4f26674eac44546e96a8d4110e5: Status 404 returned error can't find the container with id 082cbfe190b98b5a049244a3b4d2b28ab47ed4f26674eac44546e96a8d4110e5 Jul 22 16:24:58.022943 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:58.022925 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:24:58.776269 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:24:58.776237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jv9cp" event={"ID":"c8c3ca48-7d66-4816-81a2-d907c407df57","Type":"ContainerStarted","Data":"082cbfe190b98b5a049244a3b4d2b28ab47ed4f26674eac44546e96a8d4110e5"} Jul 22 16:25:03.799167 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:03.799135 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jv9cp" event={"ID":"c8c3ca48-7d66-4816-81a2-d907c407df57","Type":"ContainerStarted","Data":"2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad"} Jul 22 16:25:03.822460 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:03.822411 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-jv9cp" podStartSLOduration=1.903303743 podStartE2EDuration="6.822395772s" totalImagesPullingTime="4.919092029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:25:03.821330527 +0000 UTC m=+778.806214535" watchObservedRunningTime="2026-07-22 16:25:03.822395772 +0000 UTC m=+778.807279777" Jul 22 16:25:08.128454 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.128415 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:25:08.231987 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.231954 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:25:08.260517 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.260487 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c"] Jul 22 16:25:08.265607 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.265589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.269036 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.269016 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 22 16:25:08.269036 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.269030 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 22 16:25:08.269312 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.269295 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-nqwkl\"" Jul 22 16:25:08.269400 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.269347 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 22 16:25:08.269400 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.269386 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 22 16:25:08.287165 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.287140 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c"] Jul 22 16:25:08.353838 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.353795 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:25:08.357090 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.357069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.357161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.357104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d180bd7-32a6-47da-9028-07369f236953-feast-data\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.357198 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.357161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.357230 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.357218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-offline-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.357264 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.357254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssz2k\" (UniqueName: \"kubernetes.io/projected/7d180bd7-32a6-47da-9028-07369f236953-kube-api-access-ssz2k\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.457945 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.457875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.457945 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.457912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d180bd7-32a6-47da-9028-07369f236953-feast-data\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.458121 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:25:08.458015 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 22 16:25:08.458121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.458027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.458121 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:25:08.458073 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls podName:7d180bd7-32a6-47da-9028-07369f236953 nodeName:}" failed. No retries permitted until 2026-07-22 16:25:08.958055611 +0000 UTC m=+783.942939595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls") pod "feast-credit-scoring-7c97b4c9d4-t5g8c" (UID: "7d180bd7-32a6-47da-9028-07369f236953") : secret "feast-credit-scoring-registry-tls" not found Jul 22 16:25:08.458121 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:25:08.458117 2578 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Jul 22 16:25:08.458327 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:25:08.458156 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls podName:7d180bd7-32a6-47da-9028-07369f236953 nodeName:}" failed. No retries permitted until 2026-07-22 16:25:08.958144027 +0000 UTC m=+783.943028015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls") pod "feast-credit-scoring-7c97b4c9d4-t5g8c" (UID: "7d180bd7-32a6-47da-9028-07369f236953") : secret "feast-credit-scoring-online-tls" not found Jul 22 16:25:08.458327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.458114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-offline-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.458327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.458208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssz2k\" (UniqueName: \"kubernetes.io/projected/7d180bd7-32a6-47da-9028-07369f236953-kube-api-access-ssz2k\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.458327 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.458237 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d180bd7-32a6-47da-9028-07369f236953-feast-data\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.460357 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.460338 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-offline-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.470687 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.470665 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssz2k\" (UniqueName: \"kubernetes.io/projected/7d180bd7-32a6-47da-9028-07369f236953-kube-api-access-ssz2k\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.547420 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.547388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:25:08.700666 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.700637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:25:08.963037 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.962998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.963251 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.963081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.965570 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.965509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:08.965834 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:08.965812 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls\") pod \"feast-credit-scoring-7c97b4c9d4-t5g8c\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:09.175254 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:09.175216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:09.332961 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:09.332932 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c"] Jul 22 16:25:09.334257 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:25:09.334236 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d180bd7_32a6_47da_9028_07369f236953.slice/crio-d0a4962c48ac003e43da5db55348e28093e0a1649af5cefc1acc7da88cfe43dd WatchSource:0}: Error finding container d0a4962c48ac003e43da5db55348e28093e0a1649af5cefc1acc7da88cfe43dd: Status 404 returned error can't find the container with id d0a4962c48ac003e43da5db55348e28093e0a1649af5cefc1acc7da88cfe43dd Jul 22 16:25:09.825802 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:09.825761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerStarted","Data":"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68"} Jul 22 16:25:09.825997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:09.825863 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerStarted","Data":"d0a4962c48ac003e43da5db55348e28093e0a1649af5cefc1acc7da88cfe43dd"} Jul 22 16:25:10.831215 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:10.831177 2578 generic.go:356] "Generic (PLEG): container finished" podID="7d180bd7-32a6-47da-9028-07369f236953" containerID="844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68" exitCode=0 Jul 22 16:25:10.831638 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:10.831261 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerDied","Data":"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68"} Jul 22 16:25:11.837650 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:11.837618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerStarted","Data":"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b"} Jul 22 16:25:17.862850 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:17.862815 2578 generic.go:356] "Generic (PLEG): container finished" podID="7d180bd7-32a6-47da-9028-07369f236953" containerID="ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b" exitCode=0 Jul 22 16:25:17.863207 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:17.862855 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerDied","Data":"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b"} Jul 22 16:25:18.872004 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:18.871964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerStarted","Data":"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b"} Jul 22 16:25:18.872490 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:18.872041 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerStarted","Data":"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316"} Jul 22 16:25:18.872490 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:18.872075 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerStarted","Data":"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd"} Jul 22 16:25:18.901959 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:18.901894 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podStartSLOduration=10.90187284 podStartE2EDuration="10.90187284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:25:18.901137418 +0000 UTC m=+793.886021435" watchObservedRunningTime="2026-07-22 16:25:18.90187284 +0000 UTC m=+793.886756847" Jul 22 16:25:19.175844 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:19.175753 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:19.175844 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:19.175836 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:19.176022 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:19.175863 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:21.176143 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:21.176107 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:21.176735 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:21.176186 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:21.176735 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:21.176230 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:21.177932 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:21.177896 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.30:8816: connect: connection refused" Jul 22 16:25:21.177932 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:21.177920 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 22 16:25:21.178112 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:21.177930 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Jul 22 16:25:24.175540 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:24.175484 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Jul 22 16:25:24.176556 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:24.176507 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:24.177407 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:24.177384 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:24.181121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:24.181100 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:24.908763 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:24.908730 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:27.176383 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:27.176293 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Jul 22 16:25:30.176370 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:30.176338 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:30.177172 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:30.177151 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:25:30.400581 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:25:30.400537 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:26:28.433704 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:28.433673 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:26:33.423283 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.423252 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c"] Jul 22 16:26:33.423744 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.423484 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="registry" containerID="cri-o://3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd" gracePeriod=30 Jul 22 16:26:33.423744 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.423539 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="offline" containerID="cri-o://fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b" gracePeriod=30 Jul 22 16:26:33.423744 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.423571 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" containerID="cri-o://cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316" gracePeriod=30 Jul 22 16:26:33.438711 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.438684 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jv9cp"] Jul 22 16:26:33.438853 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.438836 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-jv9cp" podUID="c8c3ca48-7d66-4816-81a2-d907c407df57" containerName="postgres" containerID="cri-o://2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad" gracePeriod=30 Jul 22 16:26:33.516572 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.516515 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:26:33.682158 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.682096 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:26:33.745060 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.745024 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg9x6\" (UniqueName: \"kubernetes.io/projected/c8c3ca48-7d66-4816-81a2-d907c407df57-kube-api-access-fg9x6\") pod \"c8c3ca48-7d66-4816-81a2-d907c407df57\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " Jul 22 16:26:33.745256 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.745163 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8c3ca48-7d66-4816-81a2-d907c407df57-postgresdata\") pod \"c8c3ca48-7d66-4816-81a2-d907c407df57\" (UID: \"c8c3ca48-7d66-4816-81a2-d907c407df57\") " Jul 22 16:26:33.747726 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.747694 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c3ca48-7d66-4816-81a2-d907c407df57-kube-api-access-fg9x6" (OuterVolumeSpecName: "kube-api-access-fg9x6") pod "c8c3ca48-7d66-4816-81a2-d907c407df57" (UID: "c8c3ca48-7d66-4816-81a2-d907c407df57"). InnerVolumeSpecName "kube-api-access-fg9x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:26:33.774987 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.774952 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c3ca48-7d66-4816-81a2-d907c407df57-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "c8c3ca48-7d66-4816-81a2-d907c407df57" (UID: "c8c3ca48-7d66-4816-81a2-d907c407df57"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:26:33.846667 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.846630 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/c8c3ca48-7d66-4816-81a2-d907c407df57-postgresdata\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:26:33.846667 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:33.846660 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9x6\" (UniqueName: \"kubernetes.io/projected/c8c3ca48-7d66-4816-81a2-d907c407df57-kube-api-access-fg9x6\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:26:34.191252 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.191224 2578 generic.go:356] "Generic (PLEG): container finished" podID="c8c3ca48-7d66-4816-81a2-d907c407df57" containerID="2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad" exitCode=0 Jul 22 16:26:34.191374 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.191264 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-jv9cp" Jul 22 16:26:34.191374 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.191312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jv9cp" event={"ID":"c8c3ca48-7d66-4816-81a2-d907c407df57","Type":"ContainerDied","Data":"2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad"} Jul 22 16:26:34.191374 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.191357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-jv9cp" event={"ID":"c8c3ca48-7d66-4816-81a2-d907c407df57","Type":"ContainerDied","Data":"082cbfe190b98b5a049244a3b4d2b28ab47ed4f26674eac44546e96a8d4110e5"} Jul 22 16:26:34.191539 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.191377 2578 scope.go:117] "RemoveContainer" containerID="2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad" Jul 22 16:26:34.194259 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.194235 2578 generic.go:356] "Generic (PLEG): container finished" podID="7d180bd7-32a6-47da-9028-07369f236953" containerID="cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316" exitCode=0 Jul 22 16:26:34.194373 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.194265 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerDied","Data":"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316"} Jul 22 16:26:34.199439 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.199423 2578 scope.go:117] "RemoveContainer" containerID="2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad" Jul 22 16:26:34.199685 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:26:34.199668 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad\": container with ID starting with 2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad not found: ID does not exist" containerID="2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad" Jul 22 16:26:34.199744 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.199691 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad"} err="failed to get container status \"2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad\": rpc error: code = NotFound desc = could not find container \"2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad\": container with ID starting with 2b8bfba9fd6e9b70655c94c085ce64308f5a53f6a94fd746744db7acb4dccfad not found: ID does not exist" Jul 22 16:26:34.217576 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.215555 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jv9cp"] Jul 22 16:26:34.221107 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.221083 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-jv9cp"] Jul 22 16:26:34.904707 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:34.904653 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 22 16:26:35.490847 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:35.490814 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c3ca48-7d66-4816-81a2-d907c407df57" path="/var/lib/kubelet/pods/c8c3ca48-7d66-4816-81a2-d907c407df57/volumes" Jul 22 16:26:44.904514 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:44.904470 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 22 16:26:54.905118 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:54.905074 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Jul 22 16:26:54.905556 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:26:54.905195 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:27:00.278236 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:00.278123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:27:04.070165 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.070141 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:27:04.199043 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.198975 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d180bd7-32a6-47da-9028-07369f236953-feast-data\") pod \"7d180bd7-32a6-47da-9028-07369f236953\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " Jul 22 16:27:04.199043 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.199015 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls\") pod \"7d180bd7-32a6-47da-9028-07369f236953\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " Jul 22 16:27:04.199195 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.199085 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-offline-tls\") pod \"7d180bd7-32a6-47da-9028-07369f236953\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " Jul 22 16:27:04.199195 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.199106 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls\") pod \"7d180bd7-32a6-47da-9028-07369f236953\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " Jul 22 16:27:04.199195 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.199130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssz2k\" (UniqueName: \"kubernetes.io/projected/7d180bd7-32a6-47da-9028-07369f236953-kube-api-access-ssz2k\") pod \"7d180bd7-32a6-47da-9028-07369f236953\" (UID: \"7d180bd7-32a6-47da-9028-07369f236953\") " Jul 22 16:27:04.201221 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.201191 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7d180bd7-32a6-47da-9028-07369f236953" (UID: "7d180bd7-32a6-47da-9028-07369f236953"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:27:04.201335 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.201212 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7d180bd7-32a6-47da-9028-07369f236953" (UID: "7d180bd7-32a6-47da-9028-07369f236953"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:27:04.201335 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.201244 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls" (OuterVolumeSpecName: "online-tls") pod "7d180bd7-32a6-47da-9028-07369f236953" (UID: "7d180bd7-32a6-47da-9028-07369f236953"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:27:04.201434 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.201414 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d180bd7-32a6-47da-9028-07369f236953-kube-api-access-ssz2k" (OuterVolumeSpecName: "kube-api-access-ssz2k") pod "7d180bd7-32a6-47da-9028-07369f236953" (UID: "7d180bd7-32a6-47da-9028-07369f236953"). InnerVolumeSpecName "kube-api-access-ssz2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:27:04.206663 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.206637 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d180bd7-32a6-47da-9028-07369f236953-feast-data" (OuterVolumeSpecName: "feast-data") pod "7d180bd7-32a6-47da-9028-07369f236953" (UID: "7d180bd7-32a6-47da-9028-07369f236953"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:27:04.300165 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.300142 2578 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-offline-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:27:04.300165 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.300160 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-registry-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:27:04.300306 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.300169 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssz2k\" (UniqueName: \"kubernetes.io/projected/7d180bd7-32a6-47da-9028-07369f236953-kube-api-access-ssz2k\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:27:04.300306 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.300178 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7d180bd7-32a6-47da-9028-07369f236953-feast-data\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:27:04.300306 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.300186 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7d180bd7-32a6-47da-9028-07369f236953-online-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:27:04.317767 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.317743 2578 generic.go:356] "Generic (PLEG): container finished" podID="7d180bd7-32a6-47da-9028-07369f236953" containerID="fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b" exitCode=137 Jul 22 16:27:04.317767 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.317767 2578 generic.go:356] "Generic (PLEG): container finished" podID="7d180bd7-32a6-47da-9028-07369f236953" containerID="3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd" exitCode=137 Jul 22 16:27:04.317909 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.317805 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" Jul 22 16:27:04.317909 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.317820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerDied","Data":"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b"} Jul 22 16:27:04.317909 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.317868 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerDied","Data":"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd"} Jul 22 16:27:04.317909 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.317887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c" event={"ID":"7d180bd7-32a6-47da-9028-07369f236953","Type":"ContainerDied","Data":"d0a4962c48ac003e43da5db55348e28093e0a1649af5cefc1acc7da88cfe43dd"} Jul 22 16:27:04.317909 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.317908 2578 scope.go:117] "RemoveContainer" containerID="fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b" Jul 22 16:27:04.326736 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.326721 2578 scope.go:117] "RemoveContainer" containerID="cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316" Jul 22 16:27:04.333712 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.333695 2578 scope.go:117] "RemoveContainer" containerID="3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd" Jul 22 16:27:04.340263 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.340246 2578 scope.go:117] "RemoveContainer" containerID="ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b" Jul 22 16:27:04.343595 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.343573 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c"] Jul 22 16:27:04.348268 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.348254 2578 scope.go:117] "RemoveContainer" containerID="844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68" Jul 22 16:27:04.348322 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.348295 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-7c97b4c9d4-t5g8c"] Jul 22 16:27:04.354723 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.354708 2578 scope.go:117] "RemoveContainer" containerID="fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b" Jul 22 16:27:04.354949 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:27:04.354934 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b\": container with ID starting with fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b not found: ID does not exist" containerID="fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b" Jul 22 16:27:04.354995 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.354957 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b"} err="failed to get container status \"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b\": rpc error: code = NotFound desc = could not find container \"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b\": container with ID starting with fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b not found: ID does not exist" Jul 22 16:27:04.354995 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.354972 2578 scope.go:117] "RemoveContainer" containerID="cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316" Jul 22 16:27:04.355172 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:27:04.355154 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316\": container with ID starting with cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316 not found: ID does not exist" containerID="cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316" Jul 22 16:27:04.355237 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.355180 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316"} err="failed to get container status \"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316\": rpc error: code = NotFound desc = could not find container \"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316\": container with ID starting with cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316 not found: ID does not exist" Jul 22 16:27:04.355237 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.355203 2578 scope.go:117] "RemoveContainer" containerID="3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd" Jul 22 16:27:04.355410 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:27:04.355392 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd\": container with ID starting with 3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd not found: ID does not exist" containerID="3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd" Jul 22 16:27:04.355449 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.355415 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd"} err="failed to get container status \"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd\": rpc error: code = NotFound desc = could not find container \"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd\": container with ID starting with 3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd not found: ID does not exist" Jul 22 16:27:04.355449 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.355427 2578 scope.go:117] "RemoveContainer" containerID="ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b" Jul 22 16:27:04.355710 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:27:04.355693 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b\": container with ID starting with ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b not found: ID does not exist" containerID="ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b" Jul 22 16:27:04.355779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.355728 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b"} err="failed to get container status \"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b\": rpc error: code = NotFound desc = could not find container \"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b\": container with ID starting with ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b not found: ID does not exist" Jul 22 16:27:04.355779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.355749 2578 scope.go:117] "RemoveContainer" containerID="844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68" Jul 22 16:27:04.355988 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:27:04.355973 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68\": container with ID starting with 844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68 not found: ID does not exist" containerID="844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68" Jul 22 16:27:04.356051 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.355995 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68"} err="failed to get container status \"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68\": rpc error: code = NotFound desc = could not find container \"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68\": container with ID starting with 844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68 not found: ID does not exist" Jul 22 16:27:04.356051 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356013 2578 scope.go:117] "RemoveContainer" containerID="fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b" Jul 22 16:27:04.356232 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356215 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b"} err="failed to get container status \"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b\": rpc error: code = NotFound desc = could not find container \"fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b\": container with ID starting with fd284139c141b4de7e226e2500f50b135571192144286cfe36f2682f3a63504b not found: ID does not exist" Jul 22 16:27:04.356302 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356232 2578 scope.go:117] "RemoveContainer" containerID="cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316" Jul 22 16:27:04.356454 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356437 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316"} err="failed to get container status \"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316\": rpc error: code = NotFound desc = could not find container \"cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316\": container with ID starting with cfc21042e5c8c561f5751d33690be718c7af6dbf8e24680889cba8e8b5ea3316 not found: ID does not exist" Jul 22 16:27:04.356499 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356455 2578 scope.go:117] "RemoveContainer" containerID="3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd" Jul 22 16:27:04.356704 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356685 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd"} err="failed to get container status \"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd\": rpc error: code = NotFound desc = could not find container \"3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd\": container with ID starting with 3bb32958bb0cd31c189c0aa422518e45179722a1e81d6e007ce1a5ce7f51aefd not found: ID does not exist" Jul 22 16:27:04.356764 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356714 2578 scope.go:117] "RemoveContainer" containerID="ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b" Jul 22 16:27:04.356918 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356899 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b"} err="failed to get container status \"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b\": rpc error: code = NotFound desc = could not find container \"ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b\": container with ID starting with ab25c7d98b3ad82a536d8c04a337814337236e5711a483881ebcbc1602d94b3b not found: ID does not exist" Jul 22 16:27:04.356974 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.356920 2578 scope.go:117] "RemoveContainer" containerID="844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68" Jul 22 16:27:04.357123 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:04.357104 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68"} err="failed to get container status \"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68\": rpc error: code = NotFound desc = could not find container \"844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68\": container with ID starting with 844a89c4f0f8163053c03095db0b98f39be8ed68410b70858a8eed79fb6d6a68 not found: ID does not exist" Jul 22 16:27:05.395398 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:05.395370 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:27:05.397005 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:05.396984 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:27:05.401858 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:05.401836 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:27:05.403501 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:05.403486 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:27:05.490697 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:05.490663 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d180bd7-32a6-47da-9028-07369f236953" path="/var/lib/kubelet/pods/7d180bd7-32a6-47da-9028-07369f236953/volumes" Jul 22 16:27:10.691185 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:10.691155 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:27:10.793757 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:10.793724 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:27:12.040150 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:12.040119 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:27:12.104855 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:12.104823 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:27:12.201547 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:27:12.201490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:28:44.061794 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:28:44.061761 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:28:59.519946 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:28:59.519913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:12.018881 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.018777 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:12.421289 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421215 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-swkxb"] Jul 22 16:30:12.421606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421586 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="offline" Jul 22 16:30:12.421658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421609 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="offline" Jul 22 16:30:12.421658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421629 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="registry" Jul 22 16:30:12.421658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421638 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="registry" Jul 22 16:30:12.421658 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421651 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="feast-apply" Jul 22 16:30:12.421789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421660 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="feast-apply" Jul 22 16:30:12.421789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421677 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c3ca48-7d66-4816-81a2-d907c407df57" containerName="postgres" Jul 22 16:30:12.421789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421685 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c3ca48-7d66-4816-81a2-d907c407df57" containerName="postgres" Jul 22 16:30:12.421789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421694 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="feast-init" Jul 22 16:30:12.421789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421703 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="feast-init" Jul 22 16:30:12.421789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421722 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" Jul 22 16:30:12.421789 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421730 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" Jul 22 16:30:12.421993 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421811 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c3ca48-7d66-4816-81a2-d907c407df57" containerName="postgres" Jul 22 16:30:12.421993 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421825 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="online" Jul 22 16:30:12.421993 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421836 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="registry" Jul 22 16:30:12.421993 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.421846 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d180bd7-32a6-47da-9028-07369f236953" containerName="offline" Jul 22 16:30:12.425035 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.425009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:12.427793 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.427769 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 22 16:30:12.427793 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.427772 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 22 16:30:12.428796 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.428780 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-297l4\"" Jul 22 16:30:12.439169 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.439145 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-swkxb"] Jul 22 16:30:12.504911 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.504885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qvh\" (UniqueName: \"kubernetes.io/projected/901d469b-7c77-4bf0-be5e-885fab2ceb51-kube-api-access-94qvh\") pod \"redis-69bbd798b6-swkxb\" (UID: \"901d469b-7c77-4bf0-be5e-885fab2ceb51\") " pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:12.606092 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.606069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qvh\" (UniqueName: \"kubernetes.io/projected/901d469b-7c77-4bf0-be5e-885fab2ceb51-kube-api-access-94qvh\") pod \"redis-69bbd798b6-swkxb\" (UID: \"901d469b-7c77-4bf0-be5e-885fab2ceb51\") " pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:12.615714 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.615685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qvh\" (UniqueName: \"kubernetes.io/projected/901d469b-7c77-4bf0-be5e-885fab2ceb51-kube-api-access-94qvh\") pod \"redis-69bbd798b6-swkxb\" (UID: \"901d469b-7c77-4bf0-be5e-885fab2ceb51\") " pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:12.754620 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:12.754559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:14.473700 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:14.473669 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:15.340055 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:30:15.340023 2578 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 22 16:30:15.340055 ip-10-0-142-1 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_redis-69bbd798b6-swkxb_test-ns-feast-rest_901d469b-7c77-4bf0-be5e-885fab2ceb51_0(3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533): error adding pod test-ns-feast-rest_redis-69bbd798b6-swkxb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533" Netns:"/var/run/netns/dc7c838a-797e-49f0-b285-c0656cd0cd60" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=test-ns-feast-rest;K8S_POD_NAME=redis-69bbd798b6-swkxb;K8S_POD_INFRA_CONTAINER_ID=3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533;K8S_POD_UID=901d469b-7c77-4bf0-be5e-885fab2ceb51" Path:"" ERRORED: error configuring pod [test-ns-feast-rest/redis-69bbd798b6-swkxb] networking: Multus: [test-ns-feast-rest/redis-69bbd798b6-swkxb/901d469b-7c77-4bf0-be5e-885fab2ceb51]: error waiting for pod: Get "https://ae8b517e8fbe84ab18e7f53b1e50a0a7-d1dd11fbe3699cda.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/test-ns-feast-rest/pods/redis-69bbd798b6-swkxb?timeout=1m0s": context deadline exceeded Jul 22 16:30:15.340055 ip-10-0-142-1 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 22 16:30:15.340055 ip-10-0-142-1 kubenswrapper[2578]: > Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:30:15.340080 2578 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_redis-69bbd798b6-swkxb_test-ns-feast-rest_901d469b-7c77-4bf0-be5e-885fab2ceb51_0(3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533): error adding pod test-ns-feast-rest_redis-69bbd798b6-swkxb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533" Netns:"/var/run/netns/dc7c838a-797e-49f0-b285-c0656cd0cd60" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=test-ns-feast-rest;K8S_POD_NAME=redis-69bbd798b6-swkxb;K8S_POD_INFRA_CONTAINER_ID=3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533;K8S_POD_UID=901d469b-7c77-4bf0-be5e-885fab2ceb51" Path:"" ERRORED: error configuring pod [test-ns-feast-rest/redis-69bbd798b6-swkxb] networking: Multus: [test-ns-feast-rest/redis-69bbd798b6-swkxb/901d469b-7c77-4bf0-be5e-885fab2ceb51]: error waiting for pod: Get "https://ae8b517e8fbe84ab18e7f53b1e50a0a7-d1dd11fbe3699cda.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/test-ns-feast-rest/pods/redis-69bbd798b6-swkxb?timeout=1m0s": context deadline exceeded Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: > pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:30:15.340099 2578 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_redis-69bbd798b6-swkxb_test-ns-feast-rest_901d469b-7c77-4bf0-be5e-885fab2ceb51_0(3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533): error adding pod test-ns-feast-rest_redis-69bbd798b6-swkxb to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533" Netns:"/var/run/netns/dc7c838a-797e-49f0-b285-c0656cd0cd60" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=test-ns-feast-rest;K8S_POD_NAME=redis-69bbd798b6-swkxb;K8S_POD_INFRA_CONTAINER_ID=3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533;K8S_POD_UID=901d469b-7c77-4bf0-be5e-885fab2ceb51" Path:"" ERRORED: error configuring pod [test-ns-feast-rest/redis-69bbd798b6-swkxb] networking: Multus: [test-ns-feast-rest/redis-69bbd798b6-swkxb/901d469b-7c77-4bf0-be5e-885fab2ceb51]: error waiting for pod: Get "https://ae8b517e8fbe84ab18e7f53b1e50a0a7-d1dd11fbe3699cda.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/test-ns-feast-rest/pods/redis-69bbd798b6-swkxb?timeout=1m0s": context deadline exceeded Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: > pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:15.340270 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:30:15.340169 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"redis-69bbd798b6-swkxb_test-ns-feast-rest(901d469b-7c77-4bf0-be5e-885fab2ceb51)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"redis-69bbd798b6-swkxb_test-ns-feast-rest(901d469b-7c77-4bf0-be5e-885fab2ceb51)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_redis-69bbd798b6-swkxb_test-ns-feast-rest_901d469b-7c77-4bf0-be5e-885fab2ceb51_0(3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533): error adding pod test-ns-feast-rest_redis-69bbd798b6-swkxb to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533\\\" Netns:\\\"/var/run/netns/dc7c838a-797e-49f0-b285-c0656cd0cd60\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=test-ns-feast-rest;K8S_POD_NAME=redis-69bbd798b6-swkxb;K8S_POD_INFRA_CONTAINER_ID=3909776339b0d1af665c2322553a79b0fd0fe318f10473f71995bef23503c533;K8S_POD_UID=901d469b-7c77-4bf0-be5e-885fab2ceb51\\\" Path:\\\"\\\" ERRORED: error configuring pod [test-ns-feast-rest/redis-69bbd798b6-swkxb] networking: Multus: [test-ns-feast-rest/redis-69bbd798b6-swkxb/901d469b-7c77-4bf0-be5e-885fab2ceb51]: error waiting for pod: Get \\\"https://ae8b517e8fbe84ab18e7f53b1e50a0a7-d1dd11fbe3699cda.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/test-ns-feast-rest/pods/redis-69bbd798b6-swkxb?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" podUID="901d469b-7c77-4bf0-be5e-885fab2ceb51" Jul 22 16:30:16.040017 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:16.039987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:16.045908 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:16.045891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:30:16.274102 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:16.274068 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-swkxb"] Jul 22 16:30:16.277163 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:30:16.277138 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod901d469b_7c77_4bf0_be5e_885fab2ceb51.slice/crio-118acc7b402b9df07aecee71d87839c23f6db217466bda3b628ffa776c7e78a0 WatchSource:0}: Error finding container 118acc7b402b9df07aecee71d87839c23f6db217466bda3b628ffa776c7e78a0: Status 404 returned error can't find the container with id 118acc7b402b9df07aecee71d87839c23f6db217466bda3b628ffa776c7e78a0 Jul 22 16:30:16.279212 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:16.279197 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:30:17.045366 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:17.045327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" event={"ID":"901d469b-7c77-4bf0-be5e-885fab2ceb51","Type":"ContainerStarted","Data":"118acc7b402b9df07aecee71d87839c23f6db217466bda3b628ffa776c7e78a0"} Jul 22 16:30:18.050558 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:18.050510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" event={"ID":"901d469b-7c77-4bf0-be5e-885fab2ceb51","Type":"ContainerStarted","Data":"41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2"} Jul 22 16:30:18.069406 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:18.069360 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" podStartSLOduration=4.389221684 podStartE2EDuration="6.069346546s" totalImagesPullingTime="1.680124862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:30:18.067743352 +0000 UTC m=+1093.052627365" watchObservedRunningTime="2026-07-22 16:30:18.069346546 +0000 UTC m=+1093.054230551" Jul 22 16:30:23.014860 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.014819 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4ppxk"] Jul 22 16:30:23.018869 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.018846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.021699 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.021662 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 22 16:30:23.025007 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.024981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4ppxk"] Jul 22 16:30:23.061050 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.061009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:23.084404 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.084381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d69facb7-cc91-4b00-97db-3317d9dcb180-postgresdata\") pod \"postgres-864846d4f5-4ppxk\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.084491 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.084418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kr2k\" (UniqueName: \"kubernetes.io/projected/d69facb7-cc91-4b00-97db-3317d9dcb180-kube-api-access-9kr2k\") pod \"postgres-864846d4f5-4ppxk\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.185050 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.185023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d69facb7-cc91-4b00-97db-3317d9dcb180-postgresdata\") pod \"postgres-864846d4f5-4ppxk\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.185146 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.185058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kr2k\" (UniqueName: \"kubernetes.io/projected/d69facb7-cc91-4b00-97db-3317d9dcb180-kube-api-access-9kr2k\") pod \"postgres-864846d4f5-4ppxk\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.185366 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.185348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d69facb7-cc91-4b00-97db-3317d9dcb180-postgresdata\") pod \"postgres-864846d4f5-4ppxk\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.196908 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.196886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kr2k\" (UniqueName: \"kubernetes.io/projected/d69facb7-cc91-4b00-97db-3317d9dcb180-kube-api-access-9kr2k\") pod \"postgres-864846d4f5-4ppxk\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.330305 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.330252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:30:23.458721 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:23.458691 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4ppxk"] Jul 22 16:30:23.462175 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:30:23.462147 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd69facb7_cc91_4b00_97db_3317d9dcb180.slice/crio-07838e15e329b44b867e3075a9a3eff613488a3d4347ca52ca57f100a448a871 WatchSource:0}: Error finding container 07838e15e329b44b867e3075a9a3eff613488a3d4347ca52ca57f100a448a871: Status 404 returned error can't find the container with id 07838e15e329b44b867e3075a9a3eff613488a3d4347ca52ca57f100a448a871 Jul 22 16:30:24.084329 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:24.084297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" event={"ID":"d69facb7-cc91-4b00-97db-3317d9dcb180","Type":"ContainerStarted","Data":"f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84"} Jul 22 16:30:24.084701 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:24.084338 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" event={"ID":"d69facb7-cc91-4b00-97db-3317d9dcb180","Type":"ContainerStarted","Data":"07838e15e329b44b867e3075a9a3eff613488a3d4347ca52ca57f100a448a871"} Jul 22 16:30:24.101864 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:24.101810 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" podStartSLOduration=2.101793774 podStartE2EDuration="2.101793774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:30:24.101482508 +0000 UTC m=+1099.086366518" watchObservedRunningTime="2026-07-22 16:30:24.101793774 +0000 UTC m=+1099.086677780" Jul 22 16:30:28.530828 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.530778 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:28.622083 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.622058 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:28.625594 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.625569 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft"] Jul 22 16:30:28.631358 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.631340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.634285 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.634267 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 22 16:30:28.634534 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.634508 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 22 16:30:28.634740 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.634726 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-fx9gs\"" Jul 22 16:30:28.634941 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.634922 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 22 16:30:28.644751 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.644732 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft"] Jul 22 16:30:28.701101 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.701077 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:28.728805 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.728776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-online-tls\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.728904 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.728814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04b9155e-e6cc-436b-a2a8-48ba9e17b515-feast-data\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.728904 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.728851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44k9\" (UniqueName: \"kubernetes.io/projected/04b9155e-e6cc-436b-a2a8-48ba9e17b515-kube-api-access-h44k9\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.728992 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.728909 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-registry-tls\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.777023 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.776998 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:28.829846 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.829789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-online-tls\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.829846 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.829825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04b9155e-e6cc-436b-a2a8-48ba9e17b515-feast-data\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.829977 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.829848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44k9\" (UniqueName: \"kubernetes.io/projected/04b9155e-e6cc-436b-a2a8-48ba9e17b515-kube-api-access-h44k9\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.829977 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.829878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-registry-tls\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.830235 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.830215 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04b9155e-e6cc-436b-a2a8-48ba9e17b515-feast-data\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.832178 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.832158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-online-tls\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.832288 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.832271 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-registry-tls\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.841494 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.841469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44k9\" (UniqueName: \"kubernetes.io/projected/04b9155e-e6cc-436b-a2a8-48ba9e17b515-kube-api-access-h44k9\") pod \"feast-credit-scoring-56f74db96f-qwfft\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:28.940509 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:28.940487 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:29.075985 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:29.075956 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft"] Jul 22 16:30:29.078854 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:30:29.078827 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b9155e_e6cc_436b_a2a8_48ba9e17b515.slice/crio-f72716d6225f20fb622d5e95171240588e9900dbf2fec5697c23fc83e0dc82ba WatchSource:0}: Error finding container f72716d6225f20fb622d5e95171240588e9900dbf2fec5697c23fc83e0dc82ba: Status 404 returned error can't find the container with id f72716d6225f20fb622d5e95171240588e9900dbf2fec5697c23fc83e0dc82ba Jul 22 16:30:29.104586 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:29.104556 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerStarted","Data":"f72716d6225f20fb622d5e95171240588e9900dbf2fec5697c23fc83e0dc82ba"} Jul 22 16:30:30.111009 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:30.110967 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerStarted","Data":"5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894"} Jul 22 16:30:33.124384 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:33.124352 2578 generic.go:356] "Generic (PLEG): container finished" podID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerID="5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894" exitCode=0 Jul 22 16:30:33.124770 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:33.124416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerDied","Data":"5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894"} Jul 22 16:30:34.130421 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:34.130380 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerStarted","Data":"a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418"} Jul 22 16:30:40.156359 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:40.156279 2578 generic.go:356] "Generic (PLEG): container finished" podID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerID="a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418" exitCode=0 Jul 22 16:30:40.156359 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:40.156350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerDied","Data":"a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418"} Jul 22 16:30:41.164973 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:41.164939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerStarted","Data":"a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a"} Jul 22 16:30:41.165452 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:41.165003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerStarted","Data":"84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804"} Jul 22 16:30:41.189488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:41.189438 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podStartSLOduration=13.189422867 podStartE2EDuration="13.189422867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:30:41.188169586 +0000 UTC m=+1116.173053597" watchObservedRunningTime="2026-07-22 16:30:41.189422867 +0000 UTC m=+1116.174306874" Jul 22 16:30:43.941178 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:43.941097 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:43.941178 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:43.941161 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:43.942610 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:43.942577 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused" Jul 22 16:30:43.942713 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:43.942607 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 22 16:30:46.941473 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:46.941418 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused" Jul 22 16:30:46.946830 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:46.946805 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:46.947162 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:46.947142 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:46.949958 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:46.949935 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:48.941373 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:48.941328 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:49.941407 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:49.941359 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.33:6571: connect: connection refused" Jul 22 16:30:52.941803 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:52.941771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:52.942544 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:52.942510 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:30:53.261509 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:53.261483 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:54.447680 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.447634 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:54.542968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.542929 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:54.546663 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.546641 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5"] Jul 22 16:30:54.551876 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.551860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.554659 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.554639 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 22 16:30:54.554775 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.554752 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 22 16:30:54.554850 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.554800 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-6lvkr\"" Jul 22 16:30:54.554919 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.554846 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 22 16:30:54.563475 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.563453 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5"] Jul 22 16:30:54.615176 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.615146 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:54.642336 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.642310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwnl\" (UniqueName: \"kubernetes.io/projected/562a37ae-ce44-4034-902a-2ae079b9fcdc-kube-api-access-bpwnl\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.642426 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.642353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-registry-tls\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.642426 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.642404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/562a37ae-ce44-4034-902a-2ae079b9fcdc-feast-data\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.642503 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.642425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.742784 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.742729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-registry-tls\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.742784 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.742778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/562a37ae-ce44-4034-902a-2ae079b9fcdc-feast-data\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.742901 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.742812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.742901 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.742855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnl\" (UniqueName: \"kubernetes.io/projected/562a37ae-ce44-4034-902a-2ae079b9fcdc-kube-api-access-bpwnl\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.742972 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:30:54.742954 2578 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Jul 22 16:30:54.743037 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:30:54.743026 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls podName:562a37ae-ce44-4034-902a-2ae079b9fcdc nodeName:}" failed. No retries permitted until 2026-07-22 16:30:55.243007179 +0000 UTC m=+1130.227891182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls") pod "feast-driver-ranking-7f9d8b99bb-5rrj5" (UID: "562a37ae-ce44-4034-902a-2ae079b9fcdc") : secret "feast-driver-ranking-online-tls" not found Jul 22 16:30:54.743245 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.743223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/562a37ae-ce44-4034-902a-2ae079b9fcdc-feast-data\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.745243 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.745222 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-registry-tls\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.754432 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.754405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwnl\" (UniqueName: \"kubernetes.io/projected/562a37ae-ce44-4034-902a-2ae079b9fcdc-kube-api-access-bpwnl\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:54.801024 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:54.801000 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:30:55.246757 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:55.246729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:55.249097 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:55.249071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls\") pod \"feast-driver-ranking-7f9d8b99bb-5rrj5\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:55.461030 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:55.460998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:30:55.597973 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:55.597951 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5"] Jul 22 16:30:55.600242 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:30:55.600219 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562a37ae_ce44_4034_902a_2ae079b9fcdc.slice/crio-6433b7f205d1ef400e09d8b0e6d793221062e4961994754fed20ee2de72d700d WatchSource:0}: Error finding container 6433b7f205d1ef400e09d8b0e6d793221062e4961994754fed20ee2de72d700d: Status 404 returned error can't find the container with id 6433b7f205d1ef400e09d8b0e6d793221062e4961994754fed20ee2de72d700d Jul 22 16:30:56.233750 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:56.233712 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerStarted","Data":"87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3"} Jul 22 16:30:56.233956 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:30:56.233776 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerStarted","Data":"6433b7f205d1ef400e09d8b0e6d793221062e4961994754fed20ee2de72d700d"} Jul 22 16:31:00.253193 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:00.253164 2578 generic.go:356] "Generic (PLEG): container finished" podID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerID="87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3" exitCode=0 Jul 22 16:31:00.253486 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:00.253243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerDied","Data":"87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3"} Jul 22 16:31:01.259312 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:01.259280 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerStarted","Data":"496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29"} Jul 22 16:31:07.289727 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:07.289689 2578 generic.go:356] "Generic (PLEG): container finished" podID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerID="496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29" exitCode=0 Jul 22 16:31:07.290199 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:07.289769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerDied","Data":"496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29"} Jul 22 16:31:07.295405 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:07.295385 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 22 16:31:08.304682 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:08.304642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerStarted","Data":"a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07"} Jul 22 16:31:08.305065 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:08.304721 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerStarted","Data":"be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde"} Jul 22 16:31:08.332572 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:08.332495 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podStartSLOduration=14.332474463 podStartE2EDuration="14.332474463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:31:08.330858742 +0000 UTC m=+1143.315742750" watchObservedRunningTime="2026-07-22 16:31:08.332474463 +0000 UTC m=+1143.317358469" Jul 22 16:31:10.461309 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:10.461272 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:31:10.461775 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:10.461352 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:31:10.462963 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:10.462934 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 22 16:31:10.463074 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:10.462988 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 22 16:31:13.462050 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:13.462006 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 22 16:31:13.491583 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:13.491556 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:31:13.491949 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:13.491929 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:31:13.495923 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:13.495895 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:31:15.461813 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:15.461771 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:31:16.461723 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:16.461680 2578 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 22 16:31:19.462089 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:19.462058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:31:19.462852 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:31:19.462831 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:32:05.429146 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:05.429114 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:32:05.429673 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:05.429621 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:32:05.435576 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:05.435553 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:32:05.436198 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:05.436183 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:32:13.010693 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:13.010656 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:30.979095 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:30.979057 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:33.243329 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.243289 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:33.606772 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.606745 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28"] Jul 22 16:32:33.606968 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.606945 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" podUID="ab4f4fb1-ebee-4504-adaa-b36b64f7dad0" containerName="manager" containerID="cri-o://f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77" gracePeriod=10 Jul 22 16:32:33.614146 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.614122 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:33.876510 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.876480 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:32:33.882170 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.882116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:33.937756 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.937722 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft"] Jul 22 16:32:33.937991 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.937944 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="registry" containerID="cri-o://84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804" gracePeriod=30 Jul 22 16:32:33.938123 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.938028 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" containerID="cri-o://a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a" gracePeriod=30 Jul 22 16:32:33.955120 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.955087 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5"] Jul 22 16:32:33.955371 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.955345 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="registry" containerID="cri-o://be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde" gracePeriod=30 Jul 22 16:32:33.955452 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.955374 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" containerID="cri-o://a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07" gracePeriod=30 Jul 22 16:32:33.966993 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.966968 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:33.968562 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.968542 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8kqn\" (UniqueName: \"kubernetes.io/projected/ab4f4fb1-ebee-4504-adaa-b36b64f7dad0-kube-api-access-j8kqn\") pod \"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0\" (UID: \"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0\") " Jul 22 16:32:33.970537 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:33.970496 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4f4fb1-ebee-4504-adaa-b36b64f7dad0-kube-api-access-j8kqn" (OuterVolumeSpecName: "kube-api-access-j8kqn") pod "ab4f4fb1-ebee-4504-adaa-b36b64f7dad0" (UID: "ab4f4fb1-ebee-4504-adaa-b36b64f7dad0"). InnerVolumeSpecName "kube-api-access-j8kqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:32:34.069301 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.069267 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8kqn\" (UniqueName: \"kubernetes.io/projected/ab4f4fb1-ebee-4504-adaa-b36b64f7dad0-kube-api-access-j8kqn\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:32:34.672513 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.672477 2578 generic.go:356] "Generic (PLEG): container finished" podID="ab4f4fb1-ebee-4504-adaa-b36b64f7dad0" containerID="f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77" exitCode=0 Jul 22 16:32:34.672513 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.672532 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" Jul 22 16:32:34.673083 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.672592 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" event={"ID":"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0","Type":"ContainerDied","Data":"f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77"} Jul 22 16:32:34.673083 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.672648 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28" event={"ID":"ab4f4fb1-ebee-4504-adaa-b36b64f7dad0","Type":"ContainerDied","Data":"41bff6a89f84bfc0addf770ae285a906e5c33f429c2049eccb8ac90f911e23bb"} Jul 22 16:32:34.673083 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.672675 2578 scope.go:117] "RemoveContainer" containerID="f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77" Jul 22 16:32:34.684793 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.684768 2578 scope.go:117] "RemoveContainer" containerID="f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77" Jul 22 16:32:34.685419 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:32:34.685394 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77\": container with ID starting with f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77 not found: ID does not exist" containerID="f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77" Jul 22 16:32:34.685545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.685432 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77"} err="failed to get container status \"f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77\": rpc error: code = NotFound desc = could not find container \"f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77\": container with ID starting with f77124dc892f3ad5867dfe8c5e6c194601f1bee04f0843238840f5cff77bed77 not found: ID does not exist" Jul 22 16:32:34.705384 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.705298 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28"] Jul 22 16:32:34.709807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:34.709785 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-65758586b6-bzq28"] Jul 22 16:32:35.491069 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.491036 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4f4fb1-ebee-4504-adaa-b36b64f7dad0" path="/var/lib/kubelet/pods/ab4f4fb1-ebee-4504-adaa-b36b64f7dad0/volumes" Jul 22 16:32:35.678229 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.678196 2578 generic.go:356] "Generic (PLEG): container finished" podID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerID="a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a" exitCode=0 Jul 22 16:32:35.678635 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.678265 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerDied","Data":"a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a"} Jul 22 16:32:35.680428 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.680409 2578 generic.go:356] "Generic (PLEG): container finished" podID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerID="a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07" exitCode=0 Jul 22 16:32:35.680496 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.680479 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerDied","Data":"a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07"} Jul 22 16:32:35.949387 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.949350 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4ppxk"] Jul 22 16:32:35.949588 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.949537 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" podUID="d69facb7-cc91-4b00-97db-3317d9dcb180" containerName="postgres" containerID="cri-o://f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84" gracePeriod=30 Jul 22 16:32:35.961555 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.961510 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-swkxb"] Jul 22 16:32:35.961734 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:35.961699 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" podUID="901d469b-7c77-4bf0-be5e-885fab2ceb51" containerName="redis" containerID="cri-o://41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2" gracePeriod=30 Jul 22 16:32:36.030464 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.030435 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:36.216842 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.216821 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:32:36.219150 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.219133 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:32:36.386612 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.386572 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d69facb7-cc91-4b00-97db-3317d9dcb180-postgresdata\") pod \"d69facb7-cc91-4b00-97db-3317d9dcb180\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " Jul 22 16:32:36.386807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.386629 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kr2k\" (UniqueName: \"kubernetes.io/projected/d69facb7-cc91-4b00-97db-3317d9dcb180-kube-api-access-9kr2k\") pod \"d69facb7-cc91-4b00-97db-3317d9dcb180\" (UID: \"d69facb7-cc91-4b00-97db-3317d9dcb180\") " Jul 22 16:32:36.386807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.386662 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94qvh\" (UniqueName: \"kubernetes.io/projected/901d469b-7c77-4bf0-be5e-885fab2ceb51-kube-api-access-94qvh\") pod \"901d469b-7c77-4bf0-be5e-885fab2ceb51\" (UID: \"901d469b-7c77-4bf0-be5e-885fab2ceb51\") " Jul 22 16:32:36.388871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.388834 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69facb7-cc91-4b00-97db-3317d9dcb180-kube-api-access-9kr2k" (OuterVolumeSpecName: "kube-api-access-9kr2k") pod "d69facb7-cc91-4b00-97db-3317d9dcb180" (UID: "d69facb7-cc91-4b00-97db-3317d9dcb180"). InnerVolumeSpecName "kube-api-access-9kr2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:32:36.388871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.388856 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901d469b-7c77-4bf0-be5e-885fab2ceb51-kube-api-access-94qvh" (OuterVolumeSpecName: "kube-api-access-94qvh") pod "901d469b-7c77-4bf0-be5e-885fab2ceb51" (UID: "901d469b-7c77-4bf0-be5e-885fab2ceb51"). InnerVolumeSpecName "kube-api-access-94qvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:32:36.416742 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.416715 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d69facb7-cc91-4b00-97db-3317d9dcb180-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "d69facb7-cc91-4b00-97db-3317d9dcb180" (UID: "d69facb7-cc91-4b00-97db-3317d9dcb180"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:32:36.487545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.487455 2578 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/d69facb7-cc91-4b00-97db-3317d9dcb180-postgresdata\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:32:36.487545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.487482 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kr2k\" (UniqueName: \"kubernetes.io/projected/d69facb7-cc91-4b00-97db-3317d9dcb180-kube-api-access-9kr2k\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:32:36.487545 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.487494 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qvh\" (UniqueName: \"kubernetes.io/projected/901d469b-7c77-4bf0-be5e-885fab2ceb51-kube-api-access-94qvh\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:32:36.687066 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.687036 2578 generic.go:356] "Generic (PLEG): container finished" podID="901d469b-7c77-4bf0-be5e-885fab2ceb51" containerID="41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2" exitCode=0 Jul 22 16:32:36.687461 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.687078 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" Jul 22 16:32:36.687461 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.687125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" event={"ID":"901d469b-7c77-4bf0-be5e-885fab2ceb51","Type":"ContainerDied","Data":"41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2"} Jul 22 16:32:36.687461 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.687164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-swkxb" event={"ID":"901d469b-7c77-4bf0-be5e-885fab2ceb51","Type":"ContainerDied","Data":"118acc7b402b9df07aecee71d87839c23f6db217466bda3b628ffa776c7e78a0"} Jul 22 16:32:36.687461 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.687187 2578 scope.go:117] "RemoveContainer" containerID="41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2" Jul 22 16:32:36.688420 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.688401 2578 generic.go:356] "Generic (PLEG): container finished" podID="d69facb7-cc91-4b00-97db-3317d9dcb180" containerID="f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84" exitCode=0 Jul 22 16:32:36.688554 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.688443 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" Jul 22 16:32:36.688554 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.688490 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" event={"ID":"d69facb7-cc91-4b00-97db-3317d9dcb180","Type":"ContainerDied","Data":"f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84"} Jul 22 16:32:36.688554 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.688535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-4ppxk" event={"ID":"d69facb7-cc91-4b00-97db-3317d9dcb180","Type":"ContainerDied","Data":"07838e15e329b44b867e3075a9a3eff613488a3d4347ca52ca57f100a448a871"} Jul 22 16:32:36.697223 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.697200 2578 scope.go:117] "RemoveContainer" containerID="41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2" Jul 22 16:32:36.697480 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:32:36.697458 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2\": container with ID starting with 41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2 not found: ID does not exist" containerID="41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2" Jul 22 16:32:36.697606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.697489 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2"} err="failed to get container status \"41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2\": rpc error: code = NotFound desc = could not find container \"41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2\": container with ID starting with 41de97ea7098493c3ef016bf64f9211ea59bcdc856afdff8b93a2ff99edd07b2 not found: ID does not exist" Jul 22 16:32:36.697606 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.697512 2578 scope.go:117] "RemoveContainer" containerID="f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84" Jul 22 16:32:36.704503 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.704487 2578 scope.go:117] "RemoveContainer" containerID="f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84" Jul 22 16:32:36.704843 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:32:36.704826 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84\": container with ID starting with f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84 not found: ID does not exist" containerID="f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84" Jul 22 16:32:36.704898 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.704850 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84"} err="failed to get container status \"f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84\": rpc error: code = NotFound desc = could not find container \"f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84\": container with ID starting with f14f1626b65fd2673d095669bb5372d3a1c51cefc2401899413efc101adfbc84 not found: ID does not exist" Jul 22 16:32:36.712029 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.712007 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-swkxb"] Jul 22 16:32:36.716148 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.716129 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-swkxb"] Jul 22 16:32:36.728627 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.728608 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4ppxk"] Jul 22 16:32:36.735912 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.735889 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-4ppxk"] Jul 22 16:32:36.947731 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:36.947687 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 22 16:32:37.491088 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:37.491040 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901d469b-7c77-4bf0-be5e-885fab2ceb51" path="/var/lib/kubelet/pods/901d469b-7c77-4bf0-be5e-885fab2ceb51/volumes" Jul 22 16:32:37.491495 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:37.491473 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d69facb7-cc91-4b00-97db-3317d9dcb180" path="/var/lib/kubelet/pods/d69facb7-cc91-4b00-97db-3317d9dcb180/volumes" Jul 22 16:32:39.194077 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:39.194043 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:39.262740 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:39.262712 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:43.492768 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:43.492728 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 22 16:32:46.023822 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:46.023787 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:46.121913 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:46.121878 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:46.621104 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:46.621067 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:32:46.947476 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:46.947392 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 22 16:32:53.496867 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:53.496819 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 22 16:32:56.947478 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:56.947439 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" probeResult="failure" output="Get \"https://10.134.0.33:6567/health\": dial tcp 10.134.0.33:6567: connect: connection refused" Jul 22 16:32:56.947886 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:32:56.947586 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:33:02.900926 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:02.900883 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:03.002843 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.002815 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:03.316541 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.316490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:03.406906 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.406876 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:03.492405 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.492373 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 22 16:33:03.492576 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.492478 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:33:03.512676 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.512649 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:03.613491 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.613403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:03.769635 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.769605 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:03.867110 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:03.867028 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:04.587016 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.586995 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:33:04.608267 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.608243 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-registry-tls\") pod \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " Jul 22 16:33:04.608380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.608294 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h44k9\" (UniqueName: \"kubernetes.io/projected/04b9155e-e6cc-436b-a2a8-48ba9e17b515-kube-api-access-h44k9\") pod \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " Jul 22 16:33:04.608380 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.608334 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-online-tls\") pod \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " Jul 22 16:33:04.608499 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.608381 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04b9155e-e6cc-436b-a2a8-48ba9e17b515-feast-data\") pod \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\" (UID: \"04b9155e-e6cc-436b-a2a8-48ba9e17b515\") " Jul 22 16:33:04.610584 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.610556 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-online-tls" (OuterVolumeSpecName: "online-tls") pod "04b9155e-e6cc-436b-a2a8-48ba9e17b515" (UID: "04b9155e-e6cc-436b-a2a8-48ba9e17b515"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:33:04.610842 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.610815 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "04b9155e-e6cc-436b-a2a8-48ba9e17b515" (UID: "04b9155e-e6cc-436b-a2a8-48ba9e17b515"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:33:04.610842 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.610825 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b9155e-e6cc-436b-a2a8-48ba9e17b515-kube-api-access-h44k9" (OuterVolumeSpecName: "kube-api-access-h44k9") pod "04b9155e-e6cc-436b-a2a8-48ba9e17b515" (UID: "04b9155e-e6cc-436b-a2a8-48ba9e17b515"). InnerVolumeSpecName "kube-api-access-h44k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:33:04.619361 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.619336 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b9155e-e6cc-436b-a2a8-48ba9e17b515-feast-data" (OuterVolumeSpecName: "feast-data") pod "04b9155e-e6cc-436b-a2a8-48ba9e17b515" (UID: "04b9155e-e6cc-436b-a2a8-48ba9e17b515"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:33:04.620547 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.620508 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:33:04.709732 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709666 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-registry-tls\") pod \"562a37ae-ce44-4034-902a-2ae079b9fcdc\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " Jul 22 16:33:04.709732 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709697 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/562a37ae-ce44-4034-902a-2ae079b9fcdc-feast-data\") pod \"562a37ae-ce44-4034-902a-2ae079b9fcdc\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " Jul 22 16:33:04.709732 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709732 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls\") pod \"562a37ae-ce44-4034-902a-2ae079b9fcdc\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " Jul 22 16:33:04.709977 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709774 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpwnl\" (UniqueName: \"kubernetes.io/projected/562a37ae-ce44-4034-902a-2ae079b9fcdc-kube-api-access-bpwnl\") pod \"562a37ae-ce44-4034-902a-2ae079b9fcdc\" (UID: \"562a37ae-ce44-4034-902a-2ae079b9fcdc\") " Jul 22 16:33:04.709977 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709891 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-registry-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.709977 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709905 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h44k9\" (UniqueName: \"kubernetes.io/projected/04b9155e-e6cc-436b-a2a8-48ba9e17b515-kube-api-access-h44k9\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.709977 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709917 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/04b9155e-e6cc-436b-a2a8-48ba9e17b515-online-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.709977 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.709927 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/04b9155e-e6cc-436b-a2a8-48ba9e17b515-feast-data\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.710200 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.710156 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/562a37ae-ce44-4034-902a-2ae079b9fcdc-feast-data" (OuterVolumeSpecName: "feast-data") pod "562a37ae-ce44-4034-902a-2ae079b9fcdc" (UID: "562a37ae-ce44-4034-902a-2ae079b9fcdc"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:33:04.711577 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.711555 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "562a37ae-ce44-4034-902a-2ae079b9fcdc" (UID: "562a37ae-ce44-4034-902a-2ae079b9fcdc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:33:04.711680 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.711658 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls" (OuterVolumeSpecName: "online-tls") pod "562a37ae-ce44-4034-902a-2ae079b9fcdc" (UID: "562a37ae-ce44-4034-902a-2ae079b9fcdc"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:33:04.711744 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.711724 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562a37ae-ce44-4034-902a-2ae079b9fcdc-kube-api-access-bpwnl" (OuterVolumeSpecName: "kube-api-access-bpwnl") pod "562a37ae-ce44-4034-902a-2ae079b9fcdc" (UID: "562a37ae-ce44-4034-902a-2ae079b9fcdc"). InnerVolumeSpecName "kube-api-access-bpwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:33:04.806267 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.806238 2578 generic.go:356] "Generic (PLEG): container finished" podID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerID="84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804" exitCode=137 Jul 22 16:33:04.806412 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.806290 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" Jul 22 16:33:04.806412 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.806319 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerDied","Data":"84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804"} Jul 22 16:33:04.806412 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.806381 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft" event={"ID":"04b9155e-e6cc-436b-a2a8-48ba9e17b515","Type":"ContainerDied","Data":"f72716d6225f20fb622d5e95171240588e9900dbf2fec5697c23fc83e0dc82ba"} Jul 22 16:33:04.806412 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.806410 2578 scope.go:117] "RemoveContainer" containerID="a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a" Jul 22 16:33:04.808753 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.808728 2578 generic.go:356] "Generic (PLEG): container finished" podID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerID="be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde" exitCode=137 Jul 22 16:33:04.808867 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.808783 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" Jul 22 16:33:04.808867 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.808807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerDied","Data":"be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde"} Jul 22 16:33:04.808867 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.808847 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5" event={"ID":"562a37ae-ce44-4034-902a-2ae079b9fcdc","Type":"ContainerDied","Data":"6433b7f205d1ef400e09d8b0e6d793221062e4961994754fed20ee2de72d700d"} Jul 22 16:33:04.810228 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.810207 2578 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-online-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.810337 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.810229 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpwnl\" (UniqueName: \"kubernetes.io/projected/562a37ae-ce44-4034-902a-2ae079b9fcdc-kube-api-access-bpwnl\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.810337 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.810240 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/562a37ae-ce44-4034-902a-2ae079b9fcdc-registry-tls\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.810337 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.810248 2578 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/562a37ae-ce44-4034-902a-2ae079b9fcdc-feast-data\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:33:04.816951 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.816934 2578 scope.go:117] "RemoveContainer" containerID="84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804" Jul 22 16:33:04.824331 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.824314 2578 scope.go:117] "RemoveContainer" containerID="a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418" Jul 22 16:33:04.831255 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.831237 2578 scope.go:117] "RemoveContainer" containerID="5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894" Jul 22 16:33:04.831799 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.831779 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft"] Jul 22 16:33:04.836799 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.836775 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-56f74db96f-qwfft"] Jul 22 16:33:04.838430 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.838414 2578 scope.go:117] "RemoveContainer" containerID="a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a" Jul 22 16:33:04.838739 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.838719 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a\": container with ID starting with a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a not found: ID does not exist" containerID="a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a" Jul 22 16:33:04.838808 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.838749 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a"} err="failed to get container status \"a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a\": rpc error: code = NotFound desc = could not find container \"a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a\": container with ID starting with a3f0de78a6b734c417a410f669078cb09521f4f5bba8e286007f18028f44848a not found: ID does not exist" Jul 22 16:33:04.838808 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.838772 2578 scope.go:117] "RemoveContainer" containerID="84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804" Jul 22 16:33:04.839034 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.839017 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804\": container with ID starting with 84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804 not found: ID does not exist" containerID="84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804" Jul 22 16:33:04.839076 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.839038 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804"} err="failed to get container status \"84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804\": rpc error: code = NotFound desc = could not find container \"84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804\": container with ID starting with 84eac66ccd31ff8033d1310b24b67365366e0c07e1d2fb4af8ba511de747c804 not found: ID does not exist" Jul 22 16:33:04.839076 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.839053 2578 scope.go:117] "RemoveContainer" containerID="a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418" Jul 22 16:33:04.839260 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.839241 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418\": container with ID starting with a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418 not found: ID does not exist" containerID="a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418" Jul 22 16:33:04.839301 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.839265 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418"} err="failed to get container status \"a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418\": rpc error: code = NotFound desc = could not find container \"a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418\": container with ID starting with a2bad54e097a29ef4958a60116a391452a5f942e38d3295fecc8c3ed1eeff418 not found: ID does not exist" Jul 22 16:33:04.839301 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.839280 2578 scope.go:117] "RemoveContainer" containerID="5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894" Jul 22 16:33:04.839495 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.839479 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894\": container with ID starting with 5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894 not found: ID does not exist" containerID="5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894" Jul 22 16:33:04.839557 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.839498 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894"} err="failed to get container status \"5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894\": rpc error: code = NotFound desc = could not find container \"5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894\": container with ID starting with 5f37697b84bc3251b9156168a4dbdfc581560a668c17c0e03bf050284facd894 not found: ID does not exist" Jul 22 16:33:04.839557 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.839540 2578 scope.go:117] "RemoveContainer" containerID="a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07" Jul 22 16:33:04.846272 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.846257 2578 scope.go:117] "RemoveContainer" containerID="be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde" Jul 22 16:33:04.853411 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.853396 2578 scope.go:117] "RemoveContainer" containerID="496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29" Jul 22 16:33:04.853930 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.853911 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5"] Jul 22 16:33:04.860125 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.860103 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f9d8b99bb-5rrj5"] Jul 22 16:33:04.860531 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.860508 2578 scope.go:117] "RemoveContainer" containerID="87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3" Jul 22 16:33:04.867028 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.867013 2578 scope.go:117] "RemoveContainer" containerID="a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07" Jul 22 16:33:04.867267 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.867248 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07\": container with ID starting with a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07 not found: ID does not exist" containerID="a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07" Jul 22 16:33:04.867325 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.867274 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07"} err="failed to get container status \"a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07\": rpc error: code = NotFound desc = could not find container \"a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07\": container with ID starting with a8de98555180d5d089a1ff22ad9e559961b14c8cbc1723f4e88ba86837564b07 not found: ID does not exist" Jul 22 16:33:04.867325 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.867290 2578 scope.go:117] "RemoveContainer" containerID="be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde" Jul 22 16:33:04.867569 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.867551 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde\": container with ID starting with be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde not found: ID does not exist" containerID="be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde" Jul 22 16:33:04.867637 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.867574 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde"} err="failed to get container status \"be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde\": rpc error: code = NotFound desc = could not find container \"be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde\": container with ID starting with be4e10af2c5bc77c256143ae823d543d59f60d8781f31b0d707e2367b4b36cde not found: ID does not exist" Jul 22 16:33:04.867637 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.867589 2578 scope.go:117] "RemoveContainer" containerID="496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29" Jul 22 16:33:04.867828 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.867806 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29\": container with ID starting with 496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29 not found: ID does not exist" containerID="496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29" Jul 22 16:33:04.867876 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.867838 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29"} err="failed to get container status \"496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29\": rpc error: code = NotFound desc = could not find container \"496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29\": container with ID starting with 496c26fc30d7af7d98c4bae4e4a574f95e4325bdbd17bbc5068f11102a049a29 not found: ID does not exist" Jul 22 16:33:04.867876 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.867862 2578 scope.go:117] "RemoveContainer" containerID="87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3" Jul 22 16:33:04.868089 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:33:04.868067 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3\": container with ID starting with 87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3 not found: ID does not exist" containerID="87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3" Jul 22 16:33:04.868127 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:04.868092 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3"} err="failed to get container status \"87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3\": rpc error: code = NotFound desc = could not find container \"87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3\": container with ID starting with 87d0c736f03a949053da4436467546dde1a226cb6ba5dbe62a98d717da5b33d3 not found: ID does not exist" Jul 22 16:33:05.490454 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:05.490432 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" path="/var/lib/kubelet/pods/04b9155e-e6cc-436b-a2a8-48ba9e17b515/volumes" Jul 22 16:33:05.490957 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:05.490942 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" path="/var/lib/kubelet/pods/562a37ae-ce44-4034-902a-2ae079b9fcdc/volumes" Jul 22 16:33:13.992814 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:13.992782 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:14.058778 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:14.058748 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:33:14.183716 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:33:14.183679 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:34:38.926283 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:34:38.926192 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:34:39.026921 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:34:39.026886 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:34:39.150249 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:34:39.150212 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:34:39.252750 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:34:39.252659 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:34:39.355699 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:34:39.355668 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:34:39.438408 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:34:39.438374 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:34:39.596819 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:34:39.596786 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:36:00.725032 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:36:00.724949 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:36:02.453963 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:36:02.453928 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:36:49.855517 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:36:49.855481 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:36:53.673567 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:36:53.673515 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:36:53.777675 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:36:53.777643 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:37:05.459223 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:05.459189 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:37:05.459790 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:05.459607 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:37:05.465739 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:05.465717 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:37:05.466383 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:05.466362 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:37:12.938219 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:12.938178 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:37:22.515251 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:22.515210 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:37:22.684713 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:22.684681 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:37:22.782683 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:22.782590 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:37:22.850177 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:22.850148 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:37:27.841797 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:37:27.841718 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:05.768178 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:05.768143 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:05.893241 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:05.893206 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:11.885028 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:11.884992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:11.947754 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:11.947727 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:31.215966 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:31.215922 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:31.307057 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:31.307026 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:34.296348 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:34.296317 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:34.398796 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:34.398755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:34.519708 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:34.519664 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:34.621277 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:34.621203 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:38:34.735751 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:38:34.735722 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:39:33.307400 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:39:33.307321 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:39:44.471272 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:39:44.471237 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:39:55.811610 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:39:55.811566 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:39:55.910954 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:39:55.910884 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:39:55.980695 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:39:55.980663 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:16.495433 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:41:16.495396 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:18.442954 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:41:18.442920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:05.489027 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:05.488988 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:42:05.490772 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:05.490750 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:42:05.495909 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:05.495890 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:42:05.497285 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:05.497267 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:42:37.921980 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:37.921941 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:38.018313 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:38.018282 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:38.084807 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:38.084780 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:38.148171 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:38.148147 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:43.181583 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:42:43.181542 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:21.324161 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:21.324124 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:21.448801 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:21.448774 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:27.386035 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.385944 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:27.469815 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.469783 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:27.691619 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691538 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-96cpf/must-gather-b4qtj"] Jul 22 16:43:27.691913 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691895 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4f4fb1-ebee-4504-adaa-b36b64f7dad0" containerName="manager" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691915 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4f4fb1-ebee-4504-adaa-b36b64f7dad0" containerName="manager" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691927 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691935 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691949 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="feast-init" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691959 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="feast-init" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691973 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="feast-init" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691981 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="feast-init" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.691991 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="feast-apply" Jul 22 16:43:27.691997 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692000 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="feast-apply" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692016 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="registry" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692025 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="registry" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692045 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901d469b-7c77-4bf0-be5e-885fab2ceb51" containerName="redis" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692053 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="901d469b-7c77-4bf0-be5e-885fab2ceb51" containerName="redis" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692064 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="feast-apply" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692072 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="feast-apply" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692082 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d69facb7-cc91-4b00-97db-3317d9dcb180" containerName="postgres" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692090 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d69facb7-cc91-4b00-97db-3317d9dcb180" containerName="postgres" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692102 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692110 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692120 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="registry" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692128 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="registry" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692208 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4f4fb1-ebee-4504-adaa-b36b64f7dad0" containerName="manager" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692220 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="registry" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692232 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d69facb7-cc91-4b00-97db-3317d9dcb180" containerName="postgres" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692240 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="online" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692250 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="562a37ae-ce44-4034-902a-2ae079b9fcdc" containerName="online" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692261 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="901d469b-7c77-4bf0-be5e-885fab2ceb51" containerName="redis" Jul 22 16:43:27.692446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.692268 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b9155e-e6cc-436b-a2a8-48ba9e17b515" containerName="registry" Jul 22 16:43:27.695501 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.695480 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:27.698233 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.698201 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-96cpf\"/\"openshift-service-ca.crt\"" Jul 22 16:43:27.698233 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.698217 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-96cpf\"/\"default-dockercfg-bfjkk\"" Jul 22 16:43:27.698458 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.698444 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-96cpf\"/\"kube-root-ca.crt\"" Jul 22 16:43:27.705233 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.705207 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-96cpf/must-gather-b4qtj"] Jul 22 16:43:27.715959 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.715940 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:27.727272 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.727253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-must-gather-output\") pod \"must-gather-b4qtj\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:27.727376 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.727286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgx4\" (UniqueName: \"kubernetes.io/projected/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-kube-api-access-6qgx4\") pod \"must-gather-b4qtj\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:27.806462 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.806426 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:27.828248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.828218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-must-gather-output\") pod \"must-gather-b4qtj\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:27.828348 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.828257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgx4\" (UniqueName: \"kubernetes.io/projected/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-kube-api-access-6qgx4\") pod \"must-gather-b4qtj\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:27.828569 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.828546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-must-gather-output\") pod \"must-gather-b4qtj\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:27.837353 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:27.837334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgx4\" (UniqueName: \"kubernetes.io/projected/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-kube-api-access-6qgx4\") pod \"must-gather-b4qtj\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:28.011859 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:28.011840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:28.141322 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:28.141293 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-96cpf/must-gather-b4qtj"] Jul 22 16:43:28.143258 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:43:28.143236 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb47641f9_b2f1_4001_b43b_28b0b7e62d2e.slice/crio-512f1c35a403d34d6720607b6d26c97e239570d9a1717a1182836bd1d038d2b4 WatchSource:0}: Error finding container 512f1c35a403d34d6720607b6d26c97e239570d9a1717a1182836bd1d038d2b4: Status 404 returned error can't find the container with id 512f1c35a403d34d6720607b6d26c97e239570d9a1717a1182836bd1d038d2b4 Jul 22 16:43:28.145171 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:28.145155 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:43:28.169702 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:28.169674 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-96cpf/must-gather-b4qtj" event={"ID":"b47641f9-b2f1-4001-b43b-28b0b7e62d2e","Type":"ContainerStarted","Data":"512f1c35a403d34d6720607b6d26c97e239570d9a1717a1182836bd1d038d2b4"} Jul 22 16:43:35.205698 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:35.205656 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-96cpf/must-gather-b4qtj" event={"ID":"b47641f9-b2f1-4001-b43b-28b0b7e62d2e","Type":"ContainerStarted","Data":"92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0"} Jul 22 16:43:35.205698 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:35.205702 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-96cpf/must-gather-b4qtj" event={"ID":"b47641f9-b2f1-4001-b43b-28b0b7e62d2e","Type":"ContainerStarted","Data":"f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14"} Jul 22 16:43:44.244182 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:44.244147 2578 generic.go:356] "Generic (PLEG): container finished" podID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerID="f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14" exitCode=0 Jul 22 16:43:44.244735 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:44.244227 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-96cpf/must-gather-b4qtj" event={"ID":"b47641f9-b2f1-4001-b43b-28b0b7e62d2e","Type":"ContainerDied","Data":"f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14"} Jul 22 16:43:44.244735 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:44.244658 2578 scope.go:117] "RemoveContainer" containerID="f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14" Jul 22 16:43:44.656696 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:44.656666 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-96cpf_must-gather-b4qtj_b47641f9-b2f1-4001-b43b-28b0b7e62d2e/gather/0.log" Jul 22 16:43:45.014249 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:45.014220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:45.605334 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:45.605294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:45.668539 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:45.668486 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:47.838880 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:47.838850 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z8drf_1c68118a-8c93-478a-9780-3852ef7091ab/global-pull-secret-syncer/0.log" Jul 22 16:43:47.947647 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:47.947616 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" Jul 22 16:43:48.115517 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:48.115438 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-1.ec2.internal_d81f0a8852c1b51dc0d683c3eb97779a/haproxy/0.log" Jul 22 16:43:49.995248 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:49.995212 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-96cpf/must-gather-b4qtj"] Jul 22 16:43:49.995700 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:49.995391 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-96cpf/must-gather-b4qtj" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerName="copy" containerID="cri-o://92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0" gracePeriod=2 Jul 22 16:43:50.002700 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.002673 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-96cpf/must-gather-b4qtj"] Jul 22 16:43:50.005957 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:43:50.005922 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-96cpf\"/\"must-gather-b4qtj\": pods \"must-gather-b4qtj\" not found" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:50.224066 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.224046 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-96cpf_must-gather-b4qtj_b47641f9-b2f1-4001-b43b-28b0b7e62d2e/copy/0.log" Jul 22 16:43:50.229605 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.225859 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:50.233701 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:43:50.233671 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-b4qtj\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-96cpf\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:50.268779 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.268757 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-96cpf_must-gather-b4qtj_b47641f9-b2f1-4001-b43b-28b0b7e62d2e/copy/0.log" Jul 22 16:43:50.269095 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.269075 2578 generic.go:356] "Generic (PLEG): container finished" podID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerID="92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0" exitCode=143 Jul 22 16:43:50.269153 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.269131 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:50.269193 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.269155 2578 scope.go:117] "RemoveContainer" containerID="92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0" Jul 22 16:43:50.271828 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:43:50.271799 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-b4qtj\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-96cpf\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:50.276431 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.276412 2578 scope.go:117] "RemoveContainer" containerID="f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14" Jul 22 16:43:50.287916 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.287901 2578 scope.go:117] "RemoveContainer" containerID="92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0" Jul 22 16:43:50.288182 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:43:50.288163 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0\": container with ID starting with 92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0 not found: ID does not exist" containerID="92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0" Jul 22 16:43:50.288238 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.288198 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0"} err="failed to get container status \"92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0\": rpc error: code = NotFound desc = could not find container \"92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0\": container with ID starting with 92b9df5a4b68b76428b1989125c3d710232ca87d5300c0df55a5278ff6b553a0 not found: ID does not exist" Jul 22 16:43:50.288238 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.288214 2578 scope.go:117] "RemoveContainer" containerID="f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14" Jul 22 16:43:50.288482 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:43:50.288459 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14\": container with ID starting with f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14 not found: ID does not exist" containerID="f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14" Jul 22 16:43:50.288602 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.288486 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14"} err="failed to get container status \"f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14\": rpc error: code = NotFound desc = could not find container \"f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14\": container with ID starting with f1f1db1ae55dde0b02696cd3ecb2a0849ed0ef3ab21c832f7c88668cf08b2f14 not found: ID does not exist" Jul 22 16:43:50.310816 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.310796 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-must-gather-output\") pod \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " Jul 22 16:43:50.310886 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.310838 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qgx4\" (UniqueName: \"kubernetes.io/projected/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-kube-api-access-6qgx4\") pod \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\" (UID: \"b47641f9-b2f1-4001-b43b-28b0b7e62d2e\") " Jul 22 16:43:50.311116 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.311094 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "b47641f9-b2f1-4001-b43b-28b0b7e62d2e" (UID: "b47641f9-b2f1-4001-b43b-28b0b7e62d2e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:43:50.312917 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.312897 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-kube-api-access-6qgx4" (OuterVolumeSpecName: "kube-api-access-6qgx4") pod "b47641f9-b2f1-4001-b43b-28b0b7e62d2e" (UID: "b47641f9-b2f1-4001-b43b-28b0b7e62d2e"). InnerVolumeSpecName "kube-api-access-6qgx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:43:50.411304 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.411274 2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-must-gather-output\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:43:50.411304 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.411300 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qgx4\" (UniqueName: \"kubernetes.io/projected/b47641f9-b2f1-4001-b43b-28b0b7e62d2e-kube-api-access-6qgx4\") on node \"ip-10-0-142-1.ec2.internal\" DevicePath \"\"" Jul 22 16:43:50.579509 ip-10-0-142-1 kubenswrapper[2578]: E0722 16:43:50.579479 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-b4qtj\" is forbidden: User \"system:node:ip-10-0-142-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-96cpf\": no relationship found between node 'ip-10-0-142-1.ec2.internal' and this object" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" pod="openshift-must-gather-96cpf/must-gather-b4qtj" Jul 22 16:43:50.721069 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.721028 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:50.821176 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:50.821140 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhwcp_532eab52-496c-4fc8-aada-0282a49edc3c/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:51.490973 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.490937 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" path="/var/lib/kubelet/pods/b47641f9-b2f1-4001-b43b-28b0b7e62d2e/volumes" Jul 22 16:43:51.669759 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.669730 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aefbc532-4cd0-42d3-820c-70606655cff9/alertmanager/0.log" Jul 22 16:43:51.697935 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.697912 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aefbc532-4cd0-42d3-820c-70606655cff9/config-reloader/0.log" Jul 22 16:43:51.724246 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.724224 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aefbc532-4cd0-42d3-820c-70606655cff9/kube-rbac-proxy-web/0.log" Jul 22 16:43:51.749535 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.749449 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aefbc532-4cd0-42d3-820c-70606655cff9/kube-rbac-proxy/0.log" Jul 22 16:43:51.807811 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.807772 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aefbc532-4cd0-42d3-820c-70606655cff9/kube-rbac-proxy-metric/0.log" Jul 22 16:43:51.845047 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.845014 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aefbc532-4cd0-42d3-820c-70606655cff9/prom-label-proxy/0.log" Jul 22 16:43:51.872116 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.872077 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aefbc532-4cd0-42d3-820c-70606655cff9/init-config-reloader/0.log" Jul 22 16:43:51.916446 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:51.916426 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-zmt6t_854a7df1-e7ed-43a4-99f9-70f2c2bbdb3b/cluster-monitoring-operator/0.log" Jul 22 16:43:52.090769 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.090743 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-bjmbx_34fdaec6-98cb-4b19-9d76-e07618935db2/monitoring-plugin/0.log" Jul 22 16:43:52.313057 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.313030 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mtn7x_da299873-2d4c-4882-b765-4dae603e0ef8/node-exporter/0.log" Jul 22 16:43:52.339682 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.339653 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mtn7x_da299873-2d4c-4882-b765-4dae603e0ef8/kube-rbac-proxy/0.log" Jul 22 16:43:52.366116 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.366052 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mtn7x_da299873-2d4c-4882-b765-4dae603e0ef8/init-textfile/0.log" Jul 22 16:43:52.848347 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.848324 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d69d6bcdf-rqvb6_ec16a034-1158-4348-800f-915a01248b91/telemeter-client/0.log" Jul 22 16:43:52.879570 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.879516 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d69d6bcdf-rqvb6_ec16a034-1158-4348-800f-915a01248b91/reload/0.log" Jul 22 16:43:52.907871 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.907853 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d69d6bcdf-rqvb6_ec16a034-1158-4348-800f-915a01248b91/kube-rbac-proxy/0.log" Jul 22 16:43:52.957847 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.957824 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-749bf4c8d6-q2xhk_0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95/thanos-query/0.log" Jul 22 16:43:52.988384 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:52.988364 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-749bf4c8d6-q2xhk_0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95/kube-rbac-proxy-web/0.log" Jul 22 16:43:53.018354 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:53.018336 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-749bf4c8d6-q2xhk_0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95/kube-rbac-proxy/0.log" Jul 22 16:43:53.044010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:53.043984 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-749bf4c8d6-q2xhk_0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95/prom-label-proxy/0.log" Jul 22 16:43:53.076556 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:53.076502 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-749bf4c8d6-q2xhk_0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95/kube-rbac-proxy-rules/0.log" Jul 22 16:43:53.109502 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:53.109438 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-749bf4c8d6-q2xhk_0e76ee68-2c2d-4ea7-a505-09d3dc7c1d95/kube-rbac-proxy-metrics/0.log" Jul 22 16:43:54.707333 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:54.707271 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/2.log" Jul 22 16:43:54.712129 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:54.712108 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-smbvj_a285f20a-93c6-4fd8-9780-49d61469ef5e/console-operator/3.log" Jul 22 16:43:55.160505 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.160475 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-bkvbz_61042d59-56c3-452d-829b-b4459972c1a5/download-server/0.log" Jul 22 16:43:55.270125 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.270091 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj"] Jul 22 16:43:55.270412 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.270400 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerName="gather" Jul 22 16:43:55.270467 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.270414 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerName="gather" Jul 22 16:43:55.270467 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.270427 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerName="copy" Jul 22 16:43:55.270467 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.270432 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerName="copy" Jul 22 16:43:55.270576 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.270474 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerName="gather" Jul 22 16:43:55.270576 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.270483 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47641f9-b2f1-4001-b43b-28b0b7e62d2e" containerName="copy" Jul 22 16:43:55.275336 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.275317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.278126 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.278103 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wwm22\"/\"default-dockercfg-9z286\"" Jul 22 16:43:55.278266 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.278231 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wwm22\"/\"kube-root-ca.crt\"" Jul 22 16:43:55.278344 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.278328 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wwm22\"/\"openshift-service-ca.crt\"" Jul 22 16:43:55.281293 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.281273 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj"] Jul 22 16:43:55.351297 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.351265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-podres\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.351488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.351321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-lib-modules\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.351488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.351389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-proc\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.351488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.351424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rnp\" (UniqueName: \"kubernetes.io/projected/29f7583f-5620-47ca-87f5-d6d98e826f4a-kube-api-access-64rnp\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.351488 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.351463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-sys\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.452914 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.452821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-lib-modules\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.452914 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.452878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-proc\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.453087 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.452943 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-proc\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.453087 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.452985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rnp\" (UniqueName: \"kubernetes.io/projected/29f7583f-5620-47ca-87f5-d6d98e826f4a-kube-api-access-64rnp\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.453087 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.453037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-sys\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.453087 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.453067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-podres\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.453087 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.453072 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-sys\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.453276 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.452999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-lib-modules\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.453276 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.453158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/29f7583f-5620-47ca-87f5-d6d98e826f4a-podres\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.462881 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.462853 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rnp\" (UniqueName: \"kubernetes.io/projected/29f7583f-5620-47ca-87f5-d6d98e826f4a-kube-api-access-64rnp\") pod \"perf-node-gather-daemonset-k2ctj\" (UID: \"29f7583f-5620-47ca-87f5-d6d98e826f4a\") " pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.586381 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.586348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:55.676298 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.676269 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-7lx9x_630e27b6-d0f0-4d37-a45b-e0f43872e6c1/volume-data-source-validator/0.log" Jul 22 16:43:55.710934 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:55.710826 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj"] Jul 22 16:43:55.713585 ip-10-0-142-1 kubenswrapper[2578]: W0722 16:43:55.713557 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod29f7583f_5620_47ca_87f5_d6d98e826f4a.slice/crio-ca3662b80f97e71f8e005be2eee7aeaf2cfe9518bdc8fa6017a0fd7e4b8a0ce2 WatchSource:0}: Error finding container ca3662b80f97e71f8e005be2eee7aeaf2cfe9518bdc8fa6017a0fd7e4b8a0ce2: Status 404 returned error can't find the container with id ca3662b80f97e71f8e005be2eee7aeaf2cfe9518bdc8fa6017a0fd7e4b8a0ce2 Jul 22 16:43:56.294716 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:56.294681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" event={"ID":"29f7583f-5620-47ca-87f5-d6d98e826f4a","Type":"ContainerStarted","Data":"ea26c882f7859f31361c790f2dee75c26b77db1f33fd65d6ac1fd114da68c95e"} Jul 22 16:43:56.294927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:56.294731 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" event={"ID":"29f7583f-5620-47ca-87f5-d6d98e826f4a","Type":"ContainerStarted","Data":"ca3662b80f97e71f8e005be2eee7aeaf2cfe9518bdc8fa6017a0fd7e4b8a0ce2"} Jul 22 16:43:56.294927 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:56.294767 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:43:56.315172 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:56.315126 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" podStartSLOduration=1.315112832 podStartE2EDuration="1.315112832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:43:56.312926904 +0000 UTC m=+1911.297810921" watchObservedRunningTime="2026-07-22 16:43:56.315112832 +0000 UTC m=+1911.299996837" Jul 22 16:43:56.513667 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:56.513639 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4mrzj_4102150a-5b4f-4905-b488-b86f1c7e56fb/dns/0.log" Jul 22 16:43:56.539497 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:56.539473 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4mrzj_4102150a-5b4f-4905-b488-b86f1c7e56fb/kube-rbac-proxy/0.log" Jul 22 16:43:56.685086 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:56.685009 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hj59_0a3df486-be43-4d53-ac36-3e6ad9cfcb80/dns-node-resolver/0.log" Jul 22 16:43:57.181417 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:57.181385 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c5575ff86-n8jwq_79f81f04-e71d-4c4e-b027-620cde8cf2ba/registry/0.log" Jul 22 16:43:57.258741 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:57.258720 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wcksj_1a83e2e3-3411-44ab-bd64-6fefd51c5a3c/node-ca/0.log" Jul 22 16:43:58.387686 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:58.387657 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m56n4_2873ca4a-21f4-4440-8f1e-1eac30996495/serve-healthcheck-canary/0.log" Jul 22 16:43:58.831115 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:58.831086 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qnpq_d2ab600c-f104-4ad5-a749-f0e5ba6b540a/kube-rbac-proxy/0.log" Jul 22 16:43:58.855010 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:58.854988 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qnpq_d2ab600c-f104-4ad5-a749-f0e5ba6b540a/exporter/0.log" Jul 22 16:43:58.878455 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:43:58.878429 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qnpq_d2ab600c-f104-4ad5-a749-f0e5ba6b540a/extractor/0.log" Jul 22 16:44:02.308095 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:02.308068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wwm22/perf-node-gather-daemonset-k2ctj" Jul 22 16:44:04.924302 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:04.924273 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rns6_a2076767-84a2-4ba2-b556-30c266b278b6/kube-multus-additional-cni-plugins/0.log" Jul 22 16:44:04.951610 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:04.951586 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rns6_a2076767-84a2-4ba2-b556-30c266b278b6/egress-router-binary-copy/0.log" Jul 22 16:44:04.983355 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:04.983328 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rns6_a2076767-84a2-4ba2-b556-30c266b278b6/cni-plugins/0.log" Jul 22 16:44:05.011539 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:05.011499 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rns6_a2076767-84a2-4ba2-b556-30c266b278b6/bond-cni-plugin/0.log" Jul 22 16:44:05.037121 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:05.037043 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rns6_a2076767-84a2-4ba2-b556-30c266b278b6/routeoverride-cni/0.log" Jul 22 16:44:05.063080 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:05.063052 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rns6_a2076767-84a2-4ba2-b556-30c266b278b6/whereabouts-cni-bincopy/0.log" Jul 22 16:44:05.096212 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:05.096188 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rns6_a2076767-84a2-4ba2-b556-30c266b278b6/whereabouts-cni/0.log" Jul 22 16:44:05.530824 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:05.530796 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t46jb_937727a3-12c8-486f-b385-a19d18eef8dd/kube-multus/0.log" Jul 22 16:44:05.693690 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:05.693658 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwwp5_06184455-9451-417f-9a46-2009651babfc/network-metrics-daemon/0.log" Jul 22 16:44:05.718405 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:05.718376 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwwp5_06184455-9451-417f-9a46-2009651babfc/kube-rbac-proxy/0.log" Jul 22 16:44:06.812045 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.812011 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-controller/0.log" Jul 22 16:44:06.833485 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.833458 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/1.log" Jul 22 16:44:06.833643 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.833470 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovn-acl-logging/0.log" Jul 22 16:44:06.854666 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.854641 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/kube-rbac-proxy-node/0.log" Jul 22 16:44:06.879171 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.879151 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/kube-rbac-proxy-ovn-metrics/0.log" Jul 22 16:44:06.900972 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.900945 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/northd/0.log" Jul 22 16:44:06.924943 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.924877 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/nbdb/0.log" Jul 22 16:44:06.949456 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.949433 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/sbdb/0.log" Jul 22 16:44:06.994812 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:06.994782 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-87d4v_384883b4-4ca8-4839-8607-4f9536c3ebf2/ovnkube-controller/0.log" Jul 22 16:44:08.241225 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:08.241184 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7bc5b6bf85-phrs9_3ad34aa2-eaab-4523-85d5-73ed97adac32/check-endpoints/0.log" Jul 22 16:44:08.352496 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:08.352468 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gpxss_9969950d-f2c9-424c-9b21-1e6ffd76789d/network-check-target-container/0.log" Jul 22 16:44:09.309909 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:09.309878 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-782hj_87a162c9-a265-4c07-8948-ed9a6dc1c8cf/iptables-alerter/0.log" Jul 22 16:44:10.011324 ip-10-0-142-1 kubenswrapper[2578]: I0722 16:44:10.011297 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-g8s8h_f420df98-88df-4d05-97a6-47076aa5450e/tuned/0.log"