Jul 28 13:40:46.372629 ip-10-0-131-116 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 13:40:46.372838 ip-10-0-131-116 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 13:40:46.372854 ip-10-0-131-116 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 13:40:46.373253 ip-10-0-131-116 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 13:40:56.466137 ip-10-0-131-116 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 13:40:56.466160 ip-10-0-131-116 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c8e3501b3980402980ce7b6785d3b18c -- Jul 28 13:42:28.167833 ip-10-0-131-116 systemd[1]: Starting Kubernetes Kubelet... Jul 28 13:42:28.607947 ip-10-0-131-116 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 13:42:28.607947 ip-10-0-131-116 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 13:42:28.607947 ip-10-0-131-116 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 13:42:28.608410 ip-10-0-131-116 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 13:42:28.608410 ip-10-0-131-116 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 13:42:28.610834 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.610761 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 13:42:28.619846 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619826 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 13:42:28.619846 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619843 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 13:42:28.619846 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619848 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619851 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619855 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619858 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619861 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619864 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619867 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619870 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619873 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619876 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619878 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619881 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619883 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619886 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619888 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619891 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619893 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619897 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619902 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 13:42:28.619965 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619904 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619907 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619910 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619913 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619916 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619918 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619922 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619924 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619927 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619930 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619932 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619935 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619937 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619940 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619942 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619945 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619948 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619951 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619953 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 13:42:28.620415 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619956 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619958 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619961 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619963 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619966 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619968 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619971 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619974 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619976 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619979 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619982 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619985 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619987 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619990 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619993 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.619998 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620001 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620003 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620006 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 13:42:28.620889 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620009 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620012 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620014 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620017 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620019 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620022 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620024 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620026 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620029 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620034 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620037 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620041 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620044 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620047 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620050 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620053 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620055 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620058 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620060 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 13:42:28.621346 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620063 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620065 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620068 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620070 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620073 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620076 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620078 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620081 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620083 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620086 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620089 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620091 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620094 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620096 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620099 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620101 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620103 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620106 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620109 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620111 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 13:42:28.621817 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620114 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620803 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620811 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620814 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620817 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620820 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620822 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620825 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620828 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620830 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620833 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620836 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620839 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620843 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620847 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620850 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620853 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620855 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620859 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 13:42:28.622286 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620861 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620864 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620866 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620869 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620872 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620875 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620877 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620880 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620883 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620885 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620888 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620891 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620893 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620896 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620898 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620902 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620905 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620908 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620911 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620913 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 13:42:28.622759 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620916 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620919 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620921 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620924 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620926 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620929 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620933 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620936 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620938 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620941 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620944 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620947 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620950 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620952 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620956 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620958 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620961 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620963 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620967 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620970 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 13:42:28.623239 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620972 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620975 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620978 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620980 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620983 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620987 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620990 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620993 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620996 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.620999 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621002 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621005 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621008 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621011 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621014 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621016 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621019 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621022 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621024 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 13:42:28.623750 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621027 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621029 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621032 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621034 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621037 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621040 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621043 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621045 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621048 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621050 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621053 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621056 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621058 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621061 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621063 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621065 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621068 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621070 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621073 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 13:42:28.624209 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621075 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621078 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621080 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621424 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621428 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621431 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621434 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621437 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621440 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621442 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621445 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621448 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621450 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621453 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621456 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621459 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621461 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621464 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621467 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621470 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 13:42:28.624682 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621472 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621475 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621478 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621480 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621483 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621485 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621488 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621490 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621493 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621495 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621497 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621500 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621502 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621506 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621510 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621512 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621515 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621517 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621520 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621522 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 13:42:28.625184 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621525 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621527 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621530 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621532 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621535 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621537 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621540 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621544 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621547 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621551 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621554 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621556 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621559 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621561 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621564 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621566 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621571 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621573 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621590 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 13:42:28.625681 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621593 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621595 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621598 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621600 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621603 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621605 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621608 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621610 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621613 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621616 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621618 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621621 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621623 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621626 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621629 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621631 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621634 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621636 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621639 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 13:42:28.626141 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621642 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621644 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621648 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621652 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621655 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621659 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621661 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621664 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621667 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621669 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621673 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621676 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621678 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621681 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621683 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621686 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621688 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621691 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621694 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 13:42:28.626675 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621696 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621698 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621701 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621704 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.621706 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621779 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621788 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621800 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621806 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621812 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621817 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621823 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621828 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621832 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621834 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621838 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621841 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621844 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621847 2564 flags.go:64] FLAG: --cgroup-root="" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621850 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621853 2564 flags.go:64] FLAG: --client-ca-file="" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621856 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621859 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 13:42:28.627124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621866 2564 flags.go:64] FLAG: --cluster-domain="" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621869 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621872 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621875 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621878 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621882 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621885 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621888 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621891 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621894 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621897 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621900 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621903 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621905 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621912 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621915 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621919 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621922 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621925 2564 flags.go:64] FLAG: --enable-server="true" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621928 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621932 2564 flags.go:64] FLAG: --event-burst="100" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621935 2564 flags.go:64] FLAG: --event-qps="50" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621938 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621941 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 13:42:28.627667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621944 2564 flags.go:64] FLAG: --eviction-hard="" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621948 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621951 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621953 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621956 2564 flags.go:64] FLAG: --eviction-soft="" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621960 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621962 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621966 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621970 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621973 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621976 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621978 2564 flags.go:64] FLAG: --feature-gates="" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621982 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621985 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621988 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621991 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621994 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.621997 2564 flags.go:64] FLAG: --help="false" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622000 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622003 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622006 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622008 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622012 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 13:42:28.628290 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622015 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622019 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622022 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622025 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622027 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622031 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622034 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622036 2564 flags.go:64] FLAG: --kube-reserved="" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622039 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622042 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622045 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622048 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622050 2564 flags.go:64] FLAG: --lock-file="" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622053 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622056 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622059 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622064 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622069 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622072 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622075 2564 flags.go:64] FLAG: --logging-format="text" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622078 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622081 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622084 2564 flags.go:64] FLAG: --manifest-url="" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622087 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622091 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 13:42:28.628908 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622093 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622097 2564 flags.go:64] FLAG: --max-pods="110" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622100 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622103 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622106 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622109 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622112 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622114 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622118 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622125 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622128 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622131 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622134 2564 flags.go:64] FLAG: --pod-cidr="" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622137 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622142 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622145 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622148 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622151 2564 flags.go:64] FLAG: --port="10250" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622154 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622156 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b09d808c70ce5311" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622159 2564 flags.go:64] FLAG: --qos-reserved="" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622163 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622165 2564 flags.go:64] FLAG: --register-node="true" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622168 2564 flags.go:64] FLAG: --register-with-taints="" Jul 28 13:42:28.629523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622173 2564 flags.go:64] FLAG: --registry-burst="10" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622176 2564 flags.go:64] FLAG: --registry-qps="5" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622179 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622182 2564 flags.go:64] FLAG: --reserved-memory="" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622185 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622188 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622191 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622194 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622197 2564 flags.go:64] FLAG: --runonce="false" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622200 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622203 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622206 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622209 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622212 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622215 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622218 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622221 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622224 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622227 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622230 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622233 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622236 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622239 2564 flags.go:64] FLAG: --system-cgroups="" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622242 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622247 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 13:42:28.630124 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622250 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622253 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622257 2564 flags.go:64] FLAG: --tls-min-version="" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622259 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622276 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622280 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622283 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622288 2564 flags.go:64] FLAG: --v="2" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622291 2564 flags.go:64] FLAG: --version="false" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622296 2564 flags.go:64] FLAG: --vmodule="" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622300 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.622303 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622411 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622415 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622418 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622421 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622424 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622427 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622430 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622433 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622436 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622438 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622441 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 13:42:28.630869 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622444 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622448 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622451 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622453 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622456 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622458 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622461 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622464 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622467 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622470 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622473 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622476 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622478 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622481 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622484 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622486 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622490 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622492 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622495 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622498 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 13:42:28.631455 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622500 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622503 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622506 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622508 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622511 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622513 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622516 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622519 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622521 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622524 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622527 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622529 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622532 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622537 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622541 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622544 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622546 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622549 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622552 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 13:42:28.631958 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622554 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622557 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622559 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622562 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622564 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622567 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622570 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622572 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622587 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622591 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622593 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622596 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622598 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622601 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622604 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622608 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622612 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622615 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622618 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 13:42:28.632416 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622621 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622624 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622627 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622630 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622633 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622635 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622638 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622641 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622644 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622647 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622649 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622652 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622655 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622657 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622660 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622662 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622665 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622668 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622672 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622675 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 13:42:28.632888 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622678 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622681 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622684 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622687 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622690 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622692 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622695 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622698 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622701 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.622703 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.623480 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.630154 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.630169 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.630186 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.630192 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 13:42:28.633375 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630251 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630256 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630259 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630262 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630265 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630268 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630271 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630274 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630277 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630279 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630282 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630284 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630287 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630290 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630296 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630299 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630301 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630304 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630306 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630309 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 13:42:28.633802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630312 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630314 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630317 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630319 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630322 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630324 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630327 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630330 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630333 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630343 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630345 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630348 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630351 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630353 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630356 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630358 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630361 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630364 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630366 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630369 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 13:42:28.634279 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630371 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630374 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630377 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630379 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630382 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630384 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630387 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630390 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630392 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630396 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630400 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630402 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630405 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630408 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630410 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630413 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630415 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630418 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630421 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 13:42:28.634752 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630423 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630426 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630429 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630433 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630436 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630438 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630441 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630443 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630446 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630448 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630452 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630456 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630458 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630461 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630464 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630466 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630469 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630471 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 13:42:28.635289 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630475 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630478 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630481 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630484 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630487 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630489 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630492 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630494 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630497 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630499 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630502 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630505 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630507 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630509 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630512 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630515 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630518 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630521 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630524 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630526 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 13:42:28.635804 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630529 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630531 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.630536 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630654 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630658 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630661 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630664 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630666 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630669 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630671 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630674 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630678 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630682 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630684 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630687 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 13:42:28.636350 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630690 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630692 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630695 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630697 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630700 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630702 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630705 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630707 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630710 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630713 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630715 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630717 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630720 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630723 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630726 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630729 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630731 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630734 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630737 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630739 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 13:42:28.636756 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630742 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630744 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630747 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630749 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630752 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630754 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630757 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630759 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630762 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630764 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630767 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630769 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630772 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630774 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630777 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630780 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630782 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630785 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630788 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 13:42:28.637292 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630790 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630793 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630795 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630798 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630800 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630803 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630805 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630808 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630810 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630813 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630816 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630819 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630821 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630823 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630826 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630828 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630831 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630833 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630835 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630838 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 13:42:28.638093 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630840 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630843 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630845 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630848 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630850 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630853 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630855 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630858 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630860 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630863 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630865 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630868 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630870 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630873 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630876 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630879 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630882 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630884 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630887 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 13:42:28.638642 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630889 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630893 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630896 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630898 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630901 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630904 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630906 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630910 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:28.630913 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.630918 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.631010 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.633304 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.634981 2564 server.go:1019] "Starting client certificate rotation" Jul 28 13:42:28.639197 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.635096 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 13:42:28.639676 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.635144 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 13:42:28.639676 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.637715 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 13:42:28.643563 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.643540 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 13:42:28.655315 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.655299 2564 log.go:25] "Validated CRI v1 runtime API" Jul 28 13:42:28.657629 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.657616 2564 log.go:25] "Validated CRI v1 image API" Jul 28 13:42:28.658418 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.658400 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 13:42:28.663333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.663315 2564 fs.go:135] Filesystem UUIDs: map[2b376a1c-ccbd-4916-9412-5aa09d6f664a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 91287257-7d36-4d4d-870f-5ad2ec743d3d:/dev/nvme0n1p4] Jul 28 13:42:28.663394 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.663335 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 13:42:28.668484 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.668386 2564 manager.go:216] Machine: {Timestamp:2026-07-28 13:42:28.666971619 +0000 UTC m=+0.368514939 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100101 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec206342e1a33f94304aa8497fe2559c SystemUUID:ec206342-e1a3-3f94-304a-a8497fe2559c BootID:c8e3501b-3980-4029-80ce-7b6785d3b18c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:d2:5b:48:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:d2:5b:48:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:60:87:ab:94:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 13:42:28.668484 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.668482 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 13:42:28.668596 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.668549 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 13:42:28.670012 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.669998 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 13:42:28.672195 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.672169 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 13:42:28.672327 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.672199 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-116.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 13:42:28.673037 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.673028 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 13:42:28.673075 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.673039 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 13:42:28.673075 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.673046 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 13:42:28.673883 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.673873 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 13:42:28.675092 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.675083 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 13:42:28.678079 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678067 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 28 13:42:28.678191 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678182 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 13:42:28.678221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678208 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 28 13:42:28.678221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678217 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 13:42:28.678302 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678235 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 13:42:28.678302 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678243 2564 kubelet.go:398] "Adding apiserver pod source" Jul 28 13:42:28.678302 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678251 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 13:42:28.678541 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678530 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 13:42:28.678588 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.678547 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 13:42:28.681781 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.681756 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 13:42:28.684189 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.684173 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 13:42:28.684259 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.684228 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 13:42:28.686415 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686401 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 13:42:28.686415 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686417 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686422 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686428 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686437 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686446 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686457 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686462 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686469 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686475 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686483 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 13:42:28.686498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.686488 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 13:42:28.688199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.688187 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 13:42:28.688199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.688199 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 13:42:28.690174 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.690155 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-116.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 13:42:28.690264 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.690223 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 13:42:28.690359 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.690340 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 13:42:28.691179 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.691167 2564 server.go:1268] "Started kubelet" Jul 28 13:42:28.695649 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.691260 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 13:42:28.695728 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.695660 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 13:42:28.695790 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.691262 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 13:42:28.695938 ip-10-0-131-116 systemd[1]: Started Kubernetes Kubelet. Jul 28 13:42:28.697292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.697259 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kkcn8" Jul 28 13:42:28.698618 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.698273 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 13:42:28.700702 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.700539 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 28 13:42:28.703387 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.703368 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 13:42:28.703515 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.702423 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-116.ec2.internal.18c677e9e15ea39b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-116.ec2.internal,UID:ip-10-0-131-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-116.ec2.internal,},FirstTimestamp:2026-07-28 13:42:28.691141531 +0000 UTC m=+0.392684850,LastTimestamp:2026-07-28 13:42:28.691141531 +0000 UTC m=+0.392684850,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-116.ec2.internal,}" Jul 28 13:42:28.705692 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.705679 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kkcn8" Jul 28 13:42:28.708915 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.708899 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 13:42:28.708915 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.708903 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 13:42:28.709097 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.709077 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:28.709655 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709621 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 13:42:28.709655 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709643 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 13:42:28.709655 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709658 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 13:42:28.709835 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709706 2564 factory.go:55] Registering systemd factory Jul 28 13:42:28.709835 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709718 2564 factory.go:223] Registration of the systemd container factory successfully Jul 28 13:42:28.709835 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709798 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 28 13:42:28.709835 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709807 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 28 13:42:28.710129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709931 2564 factory.go:153] Registering CRI-O factory Jul 28 13:42:28.710129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709947 2564 factory.go:223] Registration of the crio container factory successfully Jul 28 13:42:28.710129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.709997 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 13:42:28.710129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.710015 2564 factory.go:103] Registering Raw factory Jul 28 13:42:28.710129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.710052 2564 manager.go:1196] Started watching for new ooms in manager Jul 28 13:42:28.710563 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.710373 2564 manager.go:349] Starting recovery of all containers Jul 28 13:42:28.713083 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.713056 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 13:42:28.713173 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.713127 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 13:42:28.720482 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.720324 2564 manager.go:354] Recovery completed Jul 28 13:42:28.721713 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.721501 2564 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 28 13:42:28.724535 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.724524 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 13:42:28.726244 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.726225 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientMemory" Jul 28 13:42:28.726307 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.726254 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 13:42:28.726307 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.726264 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientPID" Jul 28 13:42:28.726838 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.726812 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 13:42:28.726838 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.726831 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 13:42:28.726925 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.726834 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 13:42:28.726925 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.726895 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 28 13:42:28.727507 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.727493 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientMemory" Jul 28 13:42:28.727564 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.727516 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 13:42:28.727564 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.727527 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientPID" Jul 28 13:42:28.729781 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.729761 2564 policy_none.go:49] "None policy: Start" Jul 28 13:42:28.729781 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.729779 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 13:42:28.729913 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.729789 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 13:42:28.731858 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.731845 2564 policy_none.go:47] "Start" Jul 28 13:42:28.786452 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.775313 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 13:42:28.786452 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.776409 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 13:42:28.786452 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.776440 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 13:42:28.786452 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.776457 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 13:42:28.786452 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.776485 2564 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 28 13:42:28.786452 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.778775 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 13:42:28.809822 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.809793 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:28.873502 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.873458 2564 manager.go:337] "Starting Device Plugin manager" Jul 28 13:42:28.873502 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.873493 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 13:42:28.873502 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.873504 2564 server.go:87] "Starting device plugin registration server" Jul 28 13:42:28.873732 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.873720 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 13:42:28.873780 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.873734 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 13:42:28.873976 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.873853 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 13:42:28.873976 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.873924 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 13:42:28.873976 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.873931 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 13:42:28.874564 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.874397 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 13:42:28.874654 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.874628 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:28.877552 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.877535 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal"] Jul 28 13:42:28.877632 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.877611 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 13:42:28.879113 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.879099 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientMemory" Jul 28 13:42:28.879194 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.879128 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 13:42:28.879194 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.879142 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientPID" Jul 28 13:42:28.881438 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.881424 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 13:42:28.881561 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.881520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.881561 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.881543 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 13:42:28.882088 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.882072 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientMemory" Jul 28 13:42:28.882088 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.882085 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientMemory" Jul 28 13:42:28.882199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.882099 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 13:42:28.882199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.882105 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 13:42:28.882199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.882110 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientPID" Jul 28 13:42:28.882199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.882115 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientPID" Jul 28 13:42:28.884396 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.884383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.884445 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.884406 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 13:42:28.884966 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.884951 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientMemory" Jul 28 13:42:28.885047 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.884974 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 13:42:28.885047 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.884987 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientPID" Jul 28 13:42:28.919159 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.919139 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-116.ec2.internal\" not found" node="ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.921377 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.921362 2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-116.ec2.internal\" not found" node="ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.923497 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.923483 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-116.ec2.internal\" not found" node="ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.974266 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.974247 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 13:42:28.974911 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.974896 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientMemory" Jul 28 13:42:28.974986 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.974922 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 13:42:28.974986 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.974938 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeHasSufficientPID" Jul 28 13:42:28.974986 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.974961 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.983713 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:28.983700 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-116.ec2.internal" Jul 28 13:42:28.983759 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.983718 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-116.ec2.internal\": node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:28.999509 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:28.999493 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:29.011116 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.011095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0ba12ec8024ba86a45103d33226e1dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal\" (UID: \"c0ba12ec8024ba86a45103d33226e1dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.011195 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.011125 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0ba12ec8024ba86a45103d33226e1dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal\" (UID: \"c0ba12ec8024ba86a45103d33226e1dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.011195 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.011149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/675d2604a48b7f6d7a7c5b5b942d4938-config\") pod \"kube-apiserver-proxy-ip-10-0-131-116.ec2.internal\" (UID: \"675d2604a48b7f6d7a7c5b5b942d4938\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.100230 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:29.100192 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:29.111543 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.111524 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0ba12ec8024ba86a45103d33226e1dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal\" (UID: \"c0ba12ec8024ba86a45103d33226e1dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.111614 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.111550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0ba12ec8024ba86a45103d33226e1dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal\" (UID: \"c0ba12ec8024ba86a45103d33226e1dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.111614 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.111567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/675d2604a48b7f6d7a7c5b5b942d4938-config\") pod \"kube-apiserver-proxy-ip-10-0-131-116.ec2.internal\" (UID: \"675d2604a48b7f6d7a7c5b5b942d4938\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.111689 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.111621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0ba12ec8024ba86a45103d33226e1dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal\" (UID: \"c0ba12ec8024ba86a45103d33226e1dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.111689 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.111629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/675d2604a48b7f6d7a7c5b5b942d4938-config\") pod \"kube-apiserver-proxy-ip-10-0-131-116.ec2.internal\" (UID: \"675d2604a48b7f6d7a7c5b5b942d4938\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.111689 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.111657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0ba12ec8024ba86a45103d33226e1dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal\" (UID: \"c0ba12ec8024ba86a45103d33226e1dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.200926 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:29.200873 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:29.223328 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.223311 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.229007 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.228692 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.301689 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:29.301665 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:29.402151 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:29.402128 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:29.502689 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:29.502671 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:29.603292 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:29.603269 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-116.ec2.internal\" not found" Jul 28 13:42:29.633790 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.633767 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 13:42:29.634132 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.633894 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 13:42:29.697641 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.697622 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 13:42:29.709551 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.709530 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.709656 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.709555 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 13:42:29.715468 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.715446 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 13:37:29 +0000 UTC" deadline="2028-01-24 05:41:51.365928125 +0000 UTC" Jul 28 13:42:29.715522 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.715467 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13071h59m21.650463348s" Jul 28 13:42:29.723623 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.723608 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 13:42:29.725587 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.725568 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" Jul 28 13:42:29.729932 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.729916 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 13:42:29.736268 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.736254 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 13:42:29.750065 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.750046 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mngkk" Jul 28 13:42:29.761725 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.761670 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mngkk" Jul 28 13:42:29.776103 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:29.776073 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675d2604a48b7f6d7a7c5b5b942d4938.slice/crio-4bc4793a9c3a29a94257440d8c0a47ee2aab3c110773258ba6a32ac826eaf28a WatchSource:0}: Error finding container 4bc4793a9c3a29a94257440d8c0a47ee2aab3c110773258ba6a32ac826eaf28a: Status 404 returned error can't find the container with id 4bc4793a9c3a29a94257440d8c0a47ee2aab3c110773258ba6a32ac826eaf28a Jul 28 13:42:29.776828 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:29.776800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ba12ec8024ba86a45103d33226e1dc.slice/crio-5c2c6f0e5e9430965424f9de4a31cc456cf87e77f20bdfd190916789084d13c5 WatchSource:0}: Error finding container 5c2c6f0e5e9430965424f9de4a31cc456cf87e77f20bdfd190916789084d13c5: Status 404 returned error can't find the container with id 5c2c6f0e5e9430965424f9de4a31cc456cf87e77f20bdfd190916789084d13c5 Jul 28 13:42:29.781539 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.781526 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 13:42:29.851456 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.851439 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 13:42:29.992527 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:29.992502 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 13:42:30.679855 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.679817 2564 apiserver.go:52] "Watching apiserver" Jul 28 13:42:30.690254 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.690227 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 13:42:30.691709 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.691673 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cmjgg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc","openshift-cluster-node-tuning-operator/tuned-h2qwd","openshift-dns/node-resolver-wvn7w","openshift-image-registry/node-ca-tz84d","openshift-multus/multus-additional-cni-plugins-rqqk5","openshift-network-diagnostics/network-check-target-r6j7c","openshift-ovn-kubernetes/ovnkube-node-l5cdj","kube-system/konnectivity-agent-h4qjv","kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal","openshift-multus/multus-k7qpg","openshift-multus/network-metrics-daemon-k4k9j","openshift-network-operator/iptables-alerter-6ftcf"] Jul 28 13:42:30.694670 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.694599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:30.694805 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.694685 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:30.697064 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.696939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:30.697064 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.697016 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:30.699298 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.699274 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.701860 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.701611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.702403 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.702379 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 13:42:30.703510 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.703485 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 13:42:30.704016 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.703989 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-krmhf\"" Jul 28 13:42:30.705091 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.705072 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 13:42:30.705344 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.705325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-svjpr\"" Jul 28 13:42:30.705421 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.705342 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 13:42:30.705569 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.705538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.707523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.707344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.708747 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.708690 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 13:42:30.709128 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.708931 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 13:42:30.709946 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.709698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 13:42:30.710045 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.710005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:30.710340 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.710220 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-86vks\"" Jul 28 13:42:30.712278 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.712256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.712381 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.712336 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:30.714851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.714831 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.716823 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.716625 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 13:42:30.716823 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.716659 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 13:42:30.716967 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.716848 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n9nbd\"" Jul 28 13:42:30.718172 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.717424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.720028 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13431ed3-6ada-4d5d-b763-b30d0fd9c039-serviceca\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.720137 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-systemd\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.720137 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-cnibin\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.720137 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhv5k\" (UniqueName: \"kubernetes.io/projected/6f6e6450-cd8c-4979-90e5-06011a1d268c-kube-api-access-hhv5k\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:30.720137 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysctl-conf\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-run\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-var-lib-kubelet\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-host\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-tuned\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720681 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13431ed3-6ada-4d5d-b763-b30d0fd9c039-host\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-lib-modules\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.720729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-system-cni-dir\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b689bc81-3feb-45f2-b076-aa4278074c6e-agent-certs\") pod \"konnectivity-agent-h4qjv\" (UID: \"b689bc81-3feb-45f2-b076-aa4278074c6e\") " pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-modprobe-d\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-kubernetes\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/550041e0-5515-4eeb-98b2-23273b7c848d-hosts-file\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfccaf3f-da3c-430d-9640-c91af8e694f3-tmp\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-os-release\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720954 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.720984 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-kubelet-config\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysconfig\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhpn\" (UniqueName: \"kubernetes.io/projected/cfccaf3f-da3c-430d-9640-c91af8e694f3-kube-api-access-rvhpn\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-cni-binary-copy\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfw5\" (UniqueName: \"kubernetes.io/projected/92cf0d6f-c2e8-4494-a431-1a339f34f906-kube-api-access-2kfw5\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-dbus\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.721221 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:30.721965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krs7m\" (UniqueName: \"kubernetes.io/projected/13431ed3-6ada-4d5d-b763-b30d0fd9c039-kube-api-access-krs7m\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.721965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysctl-d\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.721965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/550041e0-5515-4eeb-98b2-23273b7c848d-tmp-dir\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.721965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b689bc81-3feb-45f2-b076-aa4278074c6e-konnectivity-ca\") pod \"konnectivity-agent-h4qjv\" (UID: \"b689bc81-3feb-45f2-b076-aa4278074c6e\") " pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:30.721965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:30.721965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-sys\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.721965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.721457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz8hc\" (UniqueName: \"kubernetes.io/projected/550041e0-5515-4eeb-98b2-23273b7c848d-kube-api-access-mz8hc\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.722841 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.722823 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.723281 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.723130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 13:42:30.723281 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.723273 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 13:42:30.728078 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.728060 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 13:42:30.728330 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.728311 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 13:42:30.728418 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.728338 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 13:42:30.728616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.728564 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 13:42:30.731764 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.730050 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 13:42:30.731764 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.730177 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqwwv\"" Jul 28 13:42:30.731764 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.730581 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 13:42:30.731764 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.730630 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 13:42:30.731764 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.730957 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 13:42:30.731764 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.731122 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 13:42:30.731764 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.731572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.732354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cwhzq\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.733320 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.733533 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.733713 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.733776 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xxnkj\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.734266 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.734344 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xmlw\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.734644 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 13:42:30.735844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.735257 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 13:42:30.736340 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.735988 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nf77m\"" Jul 28 13:42:30.764742 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.764700 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 13:37:30 +0000 UTC" deadline="2028-05-06 12:39:19.571760126 +0000 UTC" Jul 28 13:42:30.764830 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.764742 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15550h56m48.807021973s" Jul 28 13:42:30.781320 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.781277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" event={"ID":"c0ba12ec8024ba86a45103d33226e1dc","Type":"ContainerStarted","Data":"5c2c6f0e5e9430965424f9de4a31cc456cf87e77f20bdfd190916789084d13c5"} Jul 28 13:42:30.782478 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.782456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" event={"ID":"675d2604a48b7f6d7a7c5b5b942d4938","Type":"ContainerStarted","Data":"4bc4793a9c3a29a94257440d8c0a47ee2aab3c110773258ba6a32ac826eaf28a"} Jul 28 13:42:30.811673 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.811636 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 13:42:30.821991 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.821963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13431ed3-6ada-4d5d-b763-b30d0fd9c039-host\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.822094 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-lib-modules\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822094 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-system-cni-dir\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.822094 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b689bc81-3feb-45f2-b076-aa4278074c6e-agent-certs\") pod \"konnectivity-agent-h4qjv\" (UID: \"b689bc81-3feb-45f2-b076-aa4278074c6e\") " pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:30.822094 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13431ed3-6ada-4d5d-b763-b30d0fd9c039-host\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-ovnkube-script-lib\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-modprobe-d\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-kubernetes\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-lib-modules\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/550041e0-5515-4eeb-98b2-23273b7c848d-hosts-file\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.822226 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-device-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/550041e0-5515-4eeb-98b2-23273b7c848d-hosts-file\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-modprobe-d\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4k4\" (UniqueName: \"kubernetes.io/projected/ab63b546-6312-4b33-be18-8d788451865b-kube-api-access-cr4k4\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-system-cni-dir\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.822300 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.822300 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret podName:9d7a2dec-a85e-4a73-b3ad-245c59d03c26 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:31.322278112 +0000 UTC m=+3.023821432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret") pod "global-pull-secret-syncer-cmjgg" (UID: "9d7a2dec-a85e-4a73-b3ad-245c59d03c26") : object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-kubernetes\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-cni-bin\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfccaf3f-da3c-430d-9640-c91af8e694f3-tmp\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-os-release\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822452 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-kubelet-config\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-system-cni-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-hostroot\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-os-release\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-systemd-units\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-kubelet-config\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhpn\" (UniqueName: \"kubernetes.io/projected/cfccaf3f-da3c-430d-9640-c91af8e694f3-kube-api-access-rvhpn\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-cni-binary-copy\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfw5\" (UniqueName: \"kubernetes.io/projected/92cf0d6f-c2e8-4494-a431-1a339f34f906-kube-api-access-2kfw5\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-cni-multus\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.822922 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-kubelet\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-etc-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-cni-netd\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/550041e0-5515-4eeb-98b2-23273b7c848d-tmp-dir\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.822983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b689bc81-3feb-45f2-b076-aa4278074c6e-konnectivity-ca\") pod \"konnectivity-agent-h4qjv\" (UID: \"b689bc81-3feb-45f2-b076-aa4278074c6e\") " pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-etc-selinux\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66szl\" (UniqueName: \"kubernetes.io/projected/af2bf575-5596-4bc0-9fa4-b113e53c0279-kube-api-access-66szl\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab63b546-6312-4b33-be18-8d788451865b-host-slash\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5hvq\" (UniqueName: \"kubernetes.io/projected/e2964eed-97bc-4f82-a21e-66909e93e934-kube-api-access-v5hvq\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/550041e0-5515-4eeb-98b2-23273b7c848d-tmp-dir\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-cnibin\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-cni-binary-copy\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.823816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-etc-kubernetes\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-env-overrides\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823448 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2964eed-97bc-4f82-a21e-66909e93e934-ovn-node-metrics-cert\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-cni-bin\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-conf-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysctl-conf\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-run\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-var-lib-kubelet\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-tuned\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-var-lib-kubelet\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-cni-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-os-release\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysctl-conf\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bk4\" (UniqueName: \"kubernetes.io/projected/135af29a-f0e3-4c23-beb7-f428f6883fa9-kube-api-access-k8bk4\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-run\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.824616 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823598 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b689bc81-3feb-45f2-b076-aa4278074c6e-konnectivity-ca\") pod \"konnectivity-agent-h4qjv\" (UID: \"b689bc81-3feb-45f2-b076-aa4278074c6e\") " pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-kubelet\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-netns\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.823972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab63b546-6312-4b33-be18-8d788451865b-iptables-alerter-script\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-socket-dir-parent\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-k8s-cni-cncf-io\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysconfig\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-dbus\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-socket-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-registration-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysconfig\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-daemon-config\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krs7m\" (UniqueName: \"kubernetes.io/projected/13431ed3-6ada-4d5d-b763-b30d0fd9c039-kube-api-access-krs7m\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysctl-d\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-sys-fs\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.825343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-multus-certs\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92cf0d6f-c2e8-4494-a431-1a339f34f906-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-dbus\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-systemd\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-log-socket\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-ovnkube-config\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824418 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-sysctl-d\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-sys\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz8hc\" (UniqueName: \"kubernetes.io/projected/550041e0-5515-4eeb-98b2-23273b7c848d-kube-api-access-mz8hc\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824552 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-var-lib-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.824554 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.824625 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:42:31.324607595 +0000 UTC m=+3.026150903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-sys\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-ovn\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-node-log\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13431ed3-6ada-4d5d-b763-b30d0fd9c039-serviceca\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-systemd\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-cnibin\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-run-netns\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhv5k\" (UniqueName: \"kubernetes.io/projected/6f6e6450-cd8c-4979-90e5-06011a1d268c-kube-api-access-hhv5k\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-host\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cf0d6f-c2e8-4494-a431-1a339f34f906-cnibin\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.824984 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/135af29a-f0e3-4c23-beb7-f428f6883fa9-cni-binary-copy\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.825012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-slash\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.825029 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-systemd\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.825073 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfccaf3f-da3c-430d-9640-c91af8e694f3-host\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.826878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.825350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13431ed3-6ada-4d5d-b763-b30d0fd9c039-serviceca\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.827378 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.827161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfccaf3f-da3c-430d-9640-c91af8e694f3-etc-tuned\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.827609 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.827589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b689bc81-3feb-45f2-b076-aa4278074c6e-agent-certs\") pod \"konnectivity-agent-h4qjv\" (UID: \"b689bc81-3feb-45f2-b076-aa4278074c6e\") " pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:30.827970 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.827871 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfccaf3f-da3c-430d-9640-c91af8e694f3-tmp\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.844527 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.844504 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 13:42:30.844707 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.844677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhpn\" (UniqueName: \"kubernetes.io/projected/cfccaf3f-da3c-430d-9640-c91af8e694f3-kube-api-access-rvhpn\") pod \"tuned-h2qwd\" (UID: \"cfccaf3f-da3c-430d-9640-c91af8e694f3\") " pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:30.846950 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.846907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfw5\" (UniqueName: \"kubernetes.io/projected/92cf0d6f-c2e8-4494-a431-1a339f34f906-kube-api-access-2kfw5\") pod \"multus-additional-cni-plugins-rqqk5\" (UID: \"92cf0d6f-c2e8-4494-a431-1a339f34f906\") " pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:30.847196 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.847164 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 13:42:30.847196 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.847187 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 13:42:30.847368 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.847203 2564 projected.go:196] Error preparing data for projected volume kube-api-access-549v4 for pod openshift-network-diagnostics/network-check-target-r6j7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:30.847368 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:30.847275 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4 podName:24616ef5-ef24-4816-a359-07511a2e6c75 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:31.347248653 +0000 UTC m=+3.048791983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-549v4" (UniqueName: "kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4") pod "network-check-target-r6j7c" (UID: "24616ef5-ef24-4816-a359-07511a2e6c75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:30.847595 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.847562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhv5k\" (UniqueName: \"kubernetes.io/projected/6f6e6450-cd8c-4979-90e5-06011a1d268c-kube-api-access-hhv5k\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:30.848793 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.848754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krs7m\" (UniqueName: \"kubernetes.io/projected/13431ed3-6ada-4d5d-b763-b30d0fd9c039-kube-api-access-krs7m\") pod \"node-ca-tz84d\" (UID: \"13431ed3-6ada-4d5d-b763-b30d0fd9c039\") " pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:30.850780 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.850760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz8hc\" (UniqueName: \"kubernetes.io/projected/550041e0-5515-4eeb-98b2-23273b7c848d-kube-api-access-mz8hc\") pod \"node-resolver-wvn7w\" (UID: \"550041e0-5515-4eeb-98b2-23273b7c848d\") " pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:30.925802 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.925802 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-cnibin\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-etc-kubernetes\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-env-overrides\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-etc-kubernetes\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2964eed-97bc-4f82-a21e-66909e93e934-ovn-node-metrics-cert\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-cnibin\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-cni-bin\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.925987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.925990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-conf-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-cni-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926046 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-conf-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-cni-bin\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-os-release\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-cni-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926105 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-os-release\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bk4\" (UniqueName: \"kubernetes.io/projected/135af29a-f0e3-4c23-beb7-f428f6883fa9-kube-api-access-k8bk4\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-kubelet\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-netns\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab63b546-6312-4b33-be18-8d788451865b-iptables-alerter-script\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-netns\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-kubelet\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-socket-dir-parent\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-k8s-cni-cncf-io\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-socket-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-socket-dir-parent\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-registration-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.926496 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-env-overrides\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-daemon-config\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926396 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-k8s-cni-cncf-io\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-sys-fs\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-registration-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-multus-certs\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-sys-fs\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-run-multus-certs\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-systemd\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-socket-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926564 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-systemd\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-log-socket\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-ovnkube-config\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-log-socket\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-var-lib-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.927469 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-var-lib-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-ovn\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-node-log\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-run-netns\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-run-ovn\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/135af29a-f0e3-4c23-beb7-f428f6883fa9-cni-binary-copy\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-slash\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/135af29a-f0e3-4c23-beb7-f428f6883fa9-multus-daemon-config\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.926994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-ovnkube-script-lib\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-device-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4k4\" (UniqueName: \"kubernetes.io/projected/ab63b546-6312-4b33-be18-8d788451865b-kube-api-access-cr4k4\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-cni-bin\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-system-cni-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-hostroot\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-systemd-units\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-node-log\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-cni-multus\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-kubelet\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.928333 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-etc-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-cni-netd\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-cni-multus\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-etc-selinux\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927349 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-device-dir\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66szl\" (UniqueName: \"kubernetes.io/projected/af2bf575-5596-4bc0-9fa4-b113e53c0279-kube-api-access-66szl\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab63b546-6312-4b33-be18-8d788451865b-host-slash\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5hvq\" (UniqueName: \"kubernetes.io/projected/e2964eed-97bc-4f82-a21e-66909e93e934-kube-api-access-v5hvq\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/135af29a-f0e3-4c23-beb7-f428f6883fa9-cni-binary-copy\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-host-var-lib-kubelet\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-etc-openvswitch\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab63b546-6312-4b33-be18-8d788451865b-iptables-alerter-script\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-ovnkube-script-lib\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-cni-netd\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927586 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-slash\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927590 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-cni-bin\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.929249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-hostroot\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-run-netns\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/135af29a-f0e3-4c23-beb7-f428f6883fa9-system-cni-dir\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-systemd-units\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab63b546-6312-4b33-be18-8d788451865b-host-slash\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927180 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2964eed-97bc-4f82-a21e-66909e93e934-ovnkube-config\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2964eed-97bc-4f82-a21e-66909e93e934-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.927742 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af2bf575-5596-4bc0-9fa4-b113e53c0279-etc-selinux\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.930126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.928378 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2964eed-97bc-4f82-a21e-66909e93e934-ovn-node-metrics-cert\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:30.950664 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.950637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4k4\" (UniqueName: \"kubernetes.io/projected/ab63b546-6312-4b33-be18-8d788451865b-kube-api-access-cr4k4\") pod \"iptables-alerter-6ftcf\" (UID: \"ab63b546-6312-4b33-be18-8d788451865b\") " pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:30.952349 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.952328 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66szl\" (UniqueName: \"kubernetes.io/projected/af2bf575-5596-4bc0-9fa4-b113e53c0279-kube-api-access-66szl\") pod \"aws-ebs-csi-driver-node-kmgnc\" (UID: \"af2bf575-5596-4bc0-9fa4-b113e53c0279\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:30.953600 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.953575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bk4\" (UniqueName: \"kubernetes.io/projected/135af29a-f0e3-4c23-beb7-f428f6883fa9-kube-api-access-k8bk4\") pod \"multus-k7qpg\" (UID: \"135af29a-f0e3-4c23-beb7-f428f6883fa9\") " pod="openshift-multus/multus-k7qpg" Jul 28 13:42:30.956715 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:30.956692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5hvq\" (UniqueName: \"kubernetes.io/projected/e2964eed-97bc-4f82-a21e-66909e93e934-kube-api-access-v5hvq\") pod \"ovnkube-node-l5cdj\" (UID: \"e2964eed-97bc-4f82-a21e-66909e93e934\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:31.013303 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.013280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" Jul 28 13:42:31.021410 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.021392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wvn7w" Jul 28 13:42:31.035200 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.035177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tz84d" Jul 28 13:42:31.041068 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.041051 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" Jul 28 13:42:31.049746 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.049727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:31.056548 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.056528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" Jul 28 13:42:31.064711 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.064694 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6ftcf" Jul 28 13:42:31.073343 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.073293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:31.080190 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.080170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k7qpg" Jul 28 13:42:31.330314 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.330267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:31.330512 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.330355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:31.330512 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.330414 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:31.330512 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.330447 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:31.330512 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.330493 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret podName:9d7a2dec-a85e-4a73-b3ad-245c59d03c26 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:32.33046957 +0000 UTC m=+4.032012881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret") pod "global-pull-secret-syncer-cmjgg" (UID: "9d7a2dec-a85e-4a73-b3ad-245c59d03c26") : object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:31.330512 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.330515 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:42:32.330504281 +0000 UTC m=+4.032047591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:31.430697 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.430661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:31.430902 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.430873 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 13:42:31.430951 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.430914 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 13:42:31.430951 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.430928 2564 projected.go:196] Error preparing data for projected volume kube-api-access-549v4 for pod openshift-network-diagnostics/network-check-target-r6j7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:31.431017 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:31.431000 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4 podName:24616ef5-ef24-4816-a359-07511a2e6c75 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:32.430978908 +0000 UTC m=+4.132522231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-549v4" (UniqueName: "kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4") pod "network-check-target-r6j7c" (UID: "24616ef5-ef24-4816-a359-07511a2e6c75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:31.765429 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.765373 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 13:37:30 +0000 UTC" deadline="2028-05-06 10:45:35.555277382 +0000 UTC" Jul 28 13:42:31.765429 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:31.765418 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15549h3m3.789862895s" Jul 28 13:42:31.803362 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.803341 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb689bc81_3feb_45f2_b076_aa4278074c6e.slice/crio-a515eef3684e19fef872b5a21797e2604742490d4be67a61423f59c3bceb413a WatchSource:0}: Error finding container a515eef3684e19fef872b5a21797e2604742490d4be67a61423f59c3bceb413a: Status 404 returned error can't find the container with id a515eef3684e19fef872b5a21797e2604742490d4be67a61423f59c3bceb413a Jul 28 13:42:31.804440 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.804412 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550041e0_5515_4eeb_98b2_23273b7c848d.slice/crio-9ad6018126be9b24ce0f6f7ccc06695fd0077aac5626f0560b5134fe4be3db4e WatchSource:0}: Error finding container 9ad6018126be9b24ce0f6f7ccc06695fd0077aac5626f0560b5134fe4be3db4e: Status 404 returned error can't find the container with id 9ad6018126be9b24ce0f6f7ccc06695fd0077aac5626f0560b5134fe4be3db4e Jul 28 13:42:31.805169 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.805129 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfccaf3f_da3c_430d_9640_c91af8e694f3.slice/crio-40444c2d76ebb26cd42d98d6f2f10a594bb628eb872894309f688514dc1404fa WatchSource:0}: Error finding container 40444c2d76ebb26cd42d98d6f2f10a594bb628eb872894309f688514dc1404fa: Status 404 returned error can't find the container with id 40444c2d76ebb26cd42d98d6f2f10a594bb628eb872894309f688514dc1404fa Jul 28 13:42:31.806039 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.806003 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2964eed_97bc_4f82_a21e_66909e93e934.slice/crio-8c51727d5ce0a4baf1050e178c5b59ff98865d95f46d928e768984c79c8dcd7f WatchSource:0}: Error finding container 8c51727d5ce0a4baf1050e178c5b59ff98865d95f46d928e768984c79c8dcd7f: Status 404 returned error can't find the container with id 8c51727d5ce0a4baf1050e178c5b59ff98865d95f46d928e768984c79c8dcd7f Jul 28 13:42:31.806959 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.806918 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab63b546_6312_4b33_be18_8d788451865b.slice/crio-f8297a3815364d123d8907d58f513afac3f6fa2bc85ffc079ca003ffcb64cead WatchSource:0}: Error finding container f8297a3815364d123d8907d58f513afac3f6fa2bc85ffc079ca003ffcb64cead: Status 404 returned error can't find the container with id f8297a3815364d123d8907d58f513afac3f6fa2bc85ffc079ca003ffcb64cead Jul 28 13:42:31.810104 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.810071 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135af29a_f0e3_4c23_beb7_f428f6883fa9.slice/crio-e2e6fc80d5f54d9d6637a79541594b38f97e8a4a02462e13f0e703e040e9d47a WatchSource:0}: Error finding container e2e6fc80d5f54d9d6637a79541594b38f97e8a4a02462e13f0e703e040e9d47a: Status 404 returned error can't find the container with id e2e6fc80d5f54d9d6637a79541594b38f97e8a4a02462e13f0e703e040e9d47a Jul 28 13:42:31.811047 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.811024 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cf0d6f_c2e8_4494_a431_1a339f34f906.slice/crio-a0a451a943b2e43b297df7fae442a7c8a5820622950db03aa59e41aa3cbb8ebd WatchSource:0}: Error finding container a0a451a943b2e43b297df7fae442a7c8a5820622950db03aa59e41aa3cbb8ebd: Status 404 returned error can't find the container with id a0a451a943b2e43b297df7fae442a7c8a5820622950db03aa59e41aa3cbb8ebd Jul 28 13:42:31.811808 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.811701 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2bf575_5596_4bc0_9fa4_b113e53c0279.slice/crio-85b872124dd64b9887597296b9077510dc979ef25ac13c3cf16f978b542db40f WatchSource:0}: Error finding container 85b872124dd64b9887597296b9077510dc979ef25ac13c3cf16f978b542db40f: Status 404 returned error can't find the container with id 85b872124dd64b9887597296b9077510dc979ef25ac13c3cf16f978b542db40f Jul 28 13:42:31.814261 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:42:31.814238 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13431ed3_6ada_4d5d_b763_b30d0fd9c039.slice/crio-ac6f40082d47d54fcd8c9a1de09d1d1f2db36f54a390ea166688cfff1d092bc8 WatchSource:0}: Error finding container ac6f40082d47d54fcd8c9a1de09d1d1f2db36f54a390ea166688cfff1d092bc8: Status 404 returned error can't find the container with id ac6f40082d47d54fcd8c9a1de09d1d1f2db36f54a390ea166688cfff1d092bc8 Jul 28 13:42:32.338101 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.336943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:32.338101 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.337287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:32.338101 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.337432 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:32.338101 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.337500 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:42:34.337479556 +0000 UTC m=+6.039022868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:32.338101 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.337935 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:32.338101 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.337987 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret podName:9d7a2dec-a85e-4a73-b3ad-245c59d03c26 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:34.337971044 +0000 UTC m=+6.039514354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret") pod "global-pull-secret-syncer-cmjgg" (UID: "9d7a2dec-a85e-4a73-b3ad-245c59d03c26") : object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:32.439409 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.438746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:32.439409 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.438924 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 13:42:32.439409 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.438945 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 13:42:32.439409 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.438958 2564 projected.go:196] Error preparing data for projected volume kube-api-access-549v4 for pod openshift-network-diagnostics/network-check-target-r6j7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:32.439409 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.439017 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4 podName:24616ef5-ef24-4816-a359-07511a2e6c75 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:34.438997659 +0000 UTC m=+6.140540991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-549v4" (UniqueName: "kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4") pod "network-check-target-r6j7c" (UID: "24616ef5-ef24-4816-a359-07511a2e6c75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:32.780650 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.779751 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:32.780650 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.779888 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:32.780650 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.780355 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:32.780650 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.780454 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:32.780650 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.780516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:32.780650 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:32.780586 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:32.793098 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.793066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tz84d" event={"ID":"13431ed3-6ada-4d5d-b763-b30d0fd9c039","Type":"ContainerStarted","Data":"ac6f40082d47d54fcd8c9a1de09d1d1f2db36f54a390ea166688cfff1d092bc8"} Jul 28 13:42:32.810640 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.810613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" event={"ID":"af2bf575-5596-4bc0-9fa4-b113e53c0279","Type":"ContainerStarted","Data":"85b872124dd64b9887597296b9077510dc979ef25ac13c3cf16f978b542db40f"} Jul 28 13:42:32.821870 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.821843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7qpg" event={"ID":"135af29a-f0e3-4c23-beb7-f428f6883fa9","Type":"ContainerStarted","Data":"e2e6fc80d5f54d9d6637a79541594b38f97e8a4a02462e13f0e703e040e9d47a"} Jul 28 13:42:32.825899 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.825840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" event={"ID":"cfccaf3f-da3c-430d-9640-c91af8e694f3","Type":"ContainerStarted","Data":"40444c2d76ebb26cd42d98d6f2f10a594bb628eb872894309f688514dc1404fa"} Jul 28 13:42:32.828374 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.828282 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"8c51727d5ce0a4baf1050e178c5b59ff98865d95f46d928e768984c79c8dcd7f"} Jul 28 13:42:32.832376 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.832311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4qjv" event={"ID":"b689bc81-3feb-45f2-b076-aa4278074c6e","Type":"ContainerStarted","Data":"a515eef3684e19fef872b5a21797e2604742490d4be67a61423f59c3bceb413a"} Jul 28 13:42:32.845879 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.845183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" event={"ID":"675d2604a48b7f6d7a7c5b5b942d4938","Type":"ContainerStarted","Data":"87f597f7d2464cdd1e8e93d049877e915322a0374ec67228abc1cb43a48f5672"} Jul 28 13:42:32.860643 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.860566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerStarted","Data":"a0a451a943b2e43b297df7fae442a7c8a5820622950db03aa59e41aa3cbb8ebd"} Jul 28 13:42:32.864851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.864777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wvn7w" event={"ID":"550041e0-5515-4eeb-98b2-23273b7c848d","Type":"ContainerStarted","Data":"9ad6018126be9b24ce0f6f7ccc06695fd0077aac5626f0560b5134fe4be3db4e"} Jul 28 13:42:32.876086 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.876055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ftcf" event={"ID":"ab63b546-6312-4b33-be18-8d788451865b","Type":"ContainerStarted","Data":"f8297a3815364d123d8907d58f513afac3f6fa2bc85ffc079ca003ffcb64cead"} Jul 28 13:42:32.890998 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.890969 2564 generic.go:356] "Generic (PLEG): container finished" podID="c0ba12ec8024ba86a45103d33226e1dc" containerID="3f09e917db123ee6bc7832b435cbdaa0a076626f648ce23aaf5adae0cc9cc259" exitCode=0 Jul 28 13:42:32.891098 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.891009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" event={"ID":"c0ba12ec8024ba86a45103d33226e1dc","Type":"ContainerDied","Data":"3f09e917db123ee6bc7832b435cbdaa0a076626f648ce23aaf5adae0cc9cc259"} Jul 28 13:42:32.914592 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:32.914533 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-116.ec2.internal" podStartSLOduration=2.914514067 podStartE2EDuration="2.914514067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:32.871440034 +0000 UTC m=+4.572983367" watchObservedRunningTime="2026-07-28 13:42:32.914514067 +0000 UTC m=+4.616057403" Jul 28 13:42:33.914916 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:33.914861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" event={"ID":"c0ba12ec8024ba86a45103d33226e1dc","Type":"ContainerStarted","Data":"c2d2d1ad5f172f53c4a1da7eb865ad024c20e97b422d170be596d6399dfb5327"} Jul 28 13:42:34.355848 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:34.355804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:34.356036 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:34.355915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:34.356097 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.356039 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:34.356213 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.356105 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:42:38.356086557 +0000 UTC m=+10.057629886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:34.356213 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.356180 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:34.356343 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.356216 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret podName:9d7a2dec-a85e-4a73-b3ad-245c59d03c26 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:38.356203833 +0000 UTC m=+10.057747144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret") pod "global-pull-secret-syncer-cmjgg" (UID: "9d7a2dec-a85e-4a73-b3ad-245c59d03c26") : object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:34.456968 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:34.456886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:34.457160 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.457102 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 13:42:34.457160 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.457121 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 13:42:34.457160 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.457133 2564 projected.go:196] Error preparing data for projected volume kube-api-access-549v4 for pod openshift-network-diagnostics/network-check-target-r6j7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:34.457302 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.457190 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4 podName:24616ef5-ef24-4816-a359-07511a2e6c75 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:38.457171617 +0000 UTC m=+10.158714943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-549v4" (UniqueName: "kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4") pod "network-check-target-r6j7c" (UID: "24616ef5-ef24-4816-a359-07511a2e6c75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:34.779070 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:34.779038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:34.779248 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.779164 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:34.779554 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:34.779534 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:34.779672 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.779652 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:34.779726 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:34.779716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:34.779793 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:34.779781 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:36.777671 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:36.777637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:36.778186 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:36.777795 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:36.778186 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:36.777831 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:36.778186 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:36.777934 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:36.778186 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:36.778003 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:36.778186 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:36.778085 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:38.390149 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:38.390110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:38.390630 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:38.390168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:38.390630 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.390255 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:38.390630 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.390314 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret podName:9d7a2dec-a85e-4a73-b3ad-245c59d03c26 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:46.390296544 +0000 UTC m=+18.091839853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret") pod "global-pull-secret-syncer-cmjgg" (UID: "9d7a2dec-a85e-4a73-b3ad-245c59d03c26") : object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:38.390630 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.390255 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:38.390844 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.390709 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:42:46.390366724 +0000 UTC m=+18.091910044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:38.491094 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:38.491053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:38.491294 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.491275 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 13:42:38.491364 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.491305 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 13:42:38.491364 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.491318 2564 projected.go:196] Error preparing data for projected volume kube-api-access-549v4 for pod openshift-network-diagnostics/network-check-target-r6j7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:38.491470 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.491391 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4 podName:24616ef5-ef24-4816-a359-07511a2e6c75 nodeName:}" failed. No retries permitted until 2026-07-28 13:42:46.491373014 +0000 UTC m=+18.192916324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-549v4" (UniqueName: "kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4") pod "network-check-target-r6j7c" (UID: "24616ef5-ef24-4816-a359-07511a2e6c75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:38.777815 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:38.777783 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:38.778001 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:38.777894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:38.778001 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:38.777923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:38.778109 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.777895 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:38.778109 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.778049 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:38.778218 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:38.778194 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:40.776881 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:40.776843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:40.776881 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:40.776876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:40.777406 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:40.776843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:40.777406 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:40.776984 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:40.777406 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:40.777037 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:40.777406 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:40.777102 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:42.777248 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:42.777204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:42.777248 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:42.777205 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:42.777782 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:42.777330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:42.777782 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:42.777343 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:42.777782 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:42.777436 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:42.777782 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:42.777523 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:44.776712 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:44.776682 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:44.776712 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:44.776712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:44.777273 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:44.776679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:44.777273 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:44.776810 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:44.777273 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:44.776863 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:44.777273 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:44.776960 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:46.447696 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:46.447656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:46.448151 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:46.447711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:46.448151 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.447806 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:46.448151 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.447816 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:46.448151 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.447860 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret podName:9d7a2dec-a85e-4a73-b3ad-245c59d03c26 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:02.447847458 +0000 UTC m=+34.149390764 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret") pod "global-pull-secret-syncer-cmjgg" (UID: "9d7a2dec-a85e-4a73-b3ad-245c59d03c26") : object "kube-system"/"original-pull-secret" not registered Jul 28 13:42:46.448151 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.447886 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:43:02.44786832 +0000 UTC m=+34.149411626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:42:46.548722 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:46.548687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:46.548911 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.548822 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 13:42:46.548911 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.548843 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 13:42:46.548911 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.548856 2564 projected.go:196] Error preparing data for projected volume kube-api-access-549v4 for pod openshift-network-diagnostics/network-check-target-r6j7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:46.548911 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.548909 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4 podName:24616ef5-ef24-4816-a359-07511a2e6c75 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:02.548895548 +0000 UTC m=+34.250438855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-549v4" (UniqueName: "kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4") pod "network-check-target-r6j7c" (UID: "24616ef5-ef24-4816-a359-07511a2e6c75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:42:46.777508 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:46.777470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:46.777692 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:46.777470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:46.777692 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.777621 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:46.777692 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:46.777478 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:46.777865 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.777704 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:46.777865 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:46.777755 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:48.778542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:48.778502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:48.779029 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:48.778682 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:48.779029 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:48.778763 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:48.779029 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:48.778868 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:48.779029 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:48.778916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:48.779029 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:48.779000 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:50.777683 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.777352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:50.778352 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.777352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:50.778352 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:50.777765 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:50.778352 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:50.777854 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:50.778352 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.777382 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:50.778352 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:50.777944 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:50.951123 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.951098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tz84d" event={"ID":"13431ed3-6ada-4d5d-b763-b30d0fd9c039","Type":"ContainerStarted","Data":"b0a611c61ad9d76c0166f2585da2faf10c35fdfb66d5f1fddf13892f7ce58a3e"} Jul 28 13:42:50.952590 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.952545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" event={"ID":"af2bf575-5596-4bc0-9fa4-b113e53c0279","Type":"ContainerStarted","Data":"b4eded32dd1fb4c775372e69999f35dba5bff38de5283caa86a274b1eb43d1ba"} Jul 28 13:42:50.953934 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.953914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k7qpg" event={"ID":"135af29a-f0e3-4c23-beb7-f428f6883fa9","Type":"ContainerStarted","Data":"968384c3df7bedbdda3227dde2e9451563259c3ab0829b1ab23c8b002b2f8742"} Jul 28 13:42:50.955337 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.955317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" event={"ID":"cfccaf3f-da3c-430d-9640-c91af8e694f3","Type":"ContainerStarted","Data":"91cb54a5720dd31e093c6423efa20519bf863be726c36aa7bf8355df445ebe02"} Jul 28 13:42:50.957621 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.957602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"0ff38db031c11eb1d3962baf699515034cfe906acde0b0b1cccd5265a38a20fe"} Jul 28 13:42:50.957760 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.957665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"e3c7dd2a37656e840c80efabc1ee6a6a59f436d623edc3d5897f1ee057e6976c"} Jul 28 13:42:50.957760 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.957699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"848d23260d51b47308482e49e8c3233f3e2f7708ea211c4134c6515095b60dfe"} Jul 28 13:42:50.957760 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.957732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"8a4d952b6c5a6ae3b49e6ce3b4bb92a1acc52c6d760c19b3fac87615d72af231"} Jul 28 13:42:50.958866 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.958846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4qjv" event={"ID":"b689bc81-3feb-45f2-b076-aa4278074c6e","Type":"ContainerStarted","Data":"7560316be1a6e80449d27779867c646130dd85866c8326356785c898ec5c482d"} Jul 28 13:42:50.960034 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.960014 2564 generic.go:356] "Generic (PLEG): container finished" podID="92cf0d6f-c2e8-4494-a431-1a339f34f906" containerID="62982572a8cdbdfac02ed04b8aeaac6239554da014238b5d86fc96c58d7512e5" exitCode=0 Jul 28 13:42:50.960120 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.960086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerDied","Data":"62982572a8cdbdfac02ed04b8aeaac6239554da014238b5d86fc96c58d7512e5"} Jul 28 13:42:50.961323 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.961248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wvn7w" event={"ID":"550041e0-5515-4eeb-98b2-23273b7c848d","Type":"ContainerStarted","Data":"8ce520b7b952c13df64c966ffda132f14581fc37475351fa16fc61daf8cda4ec"} Jul 28 13:42:50.967054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.967014 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-116.ec2.internal" podStartSLOduration=20.967002807 podStartE2EDuration="20.967002807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:33.942924634 +0000 UTC m=+5.644467960" watchObservedRunningTime="2026-07-28 13:42:50.967002807 +0000 UTC m=+22.668546134" Jul 28 13:42:50.967388 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:50.967366 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tz84d" podStartSLOduration=3.385518674 podStartE2EDuration="21.967360743s" totalImagesPullingTime="18.581842069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:50.966778466 +0000 UTC m=+22.668321794" watchObservedRunningTime="2026-07-28 13:42:50.967360743 +0000 UTC m=+22.668904071" Jul 28 13:42:51.013257 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.013210 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k7qpg" podStartSLOduration=3.372080702 podStartE2EDuration="22.01319333s" totalImagesPullingTime="18.641112628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:51.01223526 +0000 UTC m=+22.713778587" watchObservedRunningTime="2026-07-28 13:42:51.01319333 +0000 UTC m=+22.714736658" Jul 28 13:42:51.029423 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.029368 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h4qjv" podStartSLOduration=7.452312578 podStartE2EDuration="22.029352259s" totalImagesPullingTime="14.577039681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:51.029005952 +0000 UTC m=+22.730549302" watchObservedRunningTime="2026-07-28 13:42:51.029352259 +0000 UTC m=+22.730895586" Jul 28 13:42:51.049152 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.049100 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h2qwd" podStartSLOduration=3.443479054 podStartE2EDuration="22.049083049s" totalImagesPullingTime="18.605603995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:51.048915745 +0000 UTC m=+22.750459073" watchObservedRunningTime="2026-07-28 13:42:51.049083049 +0000 UTC m=+22.750626379" Jul 28 13:42:51.066900 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.066851 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wvn7w" podStartSLOduration=3.470544286 podStartE2EDuration="22.066831005s" totalImagesPullingTime="18.596286719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:51.066177292 +0000 UTC m=+22.767720620" watchObservedRunningTime="2026-07-28 13:42:51.066831005 +0000 UTC m=+22.768374334" Jul 28 13:42:51.829340 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.829251 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 13:42:51.885152 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.885043 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T13:42:51.829271556Z","UUID":"aac32daa-a115-417b-bc45-47d944128214","Handler":null,"Name":"","Endpoint":""} Jul 28 13:42:51.886592 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.886544 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 13:42:51.886696 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.886612 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 13:42:51.965559 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.965477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" event={"ID":"af2bf575-5596-4bc0-9fa4-b113e53c0279","Type":"ContainerStarted","Data":"fed18b67c8d6316ccf86981cd4d758139e1216ff2b1b301e5790f432fc46e9e6"} Jul 28 13:42:51.971373 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.971335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"4f616a922a3f79f26b8ea2092e379b181c7daf70ed9f7c1e6e913bb1f324cb61"} Jul 28 13:42:51.971600 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.971514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"f6ad118e4325495f970cdb32abd81e7fb01a8da1034188834ff1adeda67a5cfe"} Jul 28 13:42:51.973559 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.973530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6ftcf" event={"ID":"ab63b546-6312-4b33-be18-8d788451865b","Type":"ContainerStarted","Data":"23e2df892621e0d2a7e0cf552cbfdf480d5d732fa5d4d92931db074985eaa39d"} Jul 28 13:42:51.989897 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:51.989847 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6ftcf" podStartSLOduration=4.400187801 podStartE2EDuration="22.989829785s" totalImagesPullingTime="18.589641984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:51.989073407 +0000 UTC m=+23.690616760" watchObservedRunningTime="2026-07-28 13:42:51.989829785 +0000 UTC m=+23.691373115" Jul 28 13:42:52.643342 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.643303 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:52.644102 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.644074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:52.776896 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.776864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:52.777078 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.776864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:52.777078 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:52.777000 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:52.777191 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:52.777085 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:52.777191 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.776864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:52.777191 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:52.777170 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:52.978831 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.978769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" event={"ID":"af2bf575-5596-4bc0-9fa4-b113e53c0279","Type":"ContainerStarted","Data":"a8d07710a7fb2483c935c100117905437d3015253ef0ed82ceae16b4fef712ff"} Jul 28 13:42:52.979413 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.979359 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:52.979924 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.979902 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h4qjv" Jul 28 13:42:52.998223 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:52.998168 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kmgnc" podStartSLOduration=3.2561824 podStartE2EDuration="23.998147424s" totalImagesPullingTime="20.741965024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:52.99756862 +0000 UTC m=+24.699111949" watchObservedRunningTime="2026-07-28 13:42:52.998147424 +0000 UTC m=+24.699690757" Jul 28 13:42:53.985907 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:53.985728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"5113d0288be22c80f5f8e3835bdfcbb926936a3bff9313f97b59038ff82481f0"} Jul 28 13:42:54.777601 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:54.777519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:54.777601 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:54.777550 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:54.777601 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:54.777519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:54.777939 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:54.777674 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:54.777939 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:54.777749 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:54.777939 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:54.777880 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:55.992742 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:55.992503 2564 generic.go:356] "Generic (PLEG): container finished" podID="92cf0d6f-c2e8-4494-a431-1a339f34f906" containerID="ac6881e838cb4a5948f77d761e5857a1fc944d525295b63ea328c67fd78ede0d" exitCode=0 Jul 28 13:42:55.993523 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:55.992551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerDied","Data":"ac6881e838cb4a5948f77d761e5857a1fc944d525295b63ea328c67fd78ede0d"} Jul 28 13:42:55.996415 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:55.996389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" event={"ID":"e2964eed-97bc-4f82-a21e-66909e93e934","Type":"ContainerStarted","Data":"871d29c99208c59ad10af3c49cd694ce733228aadac01c80e864a6271aa4af00"} Jul 28 13:42:55.996621 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:55.996604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:55.996675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:55.996666 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:56.012516 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:56.012489 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:56.044302 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:56.044251 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" podStartSLOduration=8.382839946 podStartE2EDuration="27.044235162s" totalImagesPullingTime="18.661395216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:42:56.043067243 +0000 UTC m=+27.744610571" watchObservedRunningTime="2026-07-28 13:42:56.044235162 +0000 UTC m=+27.745778471" Jul 28 13:42:56.776886 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:56.776853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:56.777280 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:56.776853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:56.777280 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:56.777031 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:56.777280 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:56.777035 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:56.777280 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:56.777128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:56.777280 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:56.777217 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:57.017524 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.017450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:57.033199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.033074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:42:57.746671 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.746641 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k4k9j"] Jul 28 13:42:57.746825 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.746785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:57.746894 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:57.746869 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:42:57.749763 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.749729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6j7c"] Jul 28 13:42:57.749898 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.749838 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:57.749964 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:57.749920 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:57.750570 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.750546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmjgg"] Jul 28 13:42:57.750697 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:57.750657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:57.750777 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:57.750748 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:58.002440 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:58.002408 2564 generic.go:356] "Generic (PLEG): container finished" podID="92cf0d6f-c2e8-4494-a431-1a339f34f906" containerID="a0300cda9a29d98e9bce3ac8781510184e7e34fe1686a626e3b8ae14374a8ee1" exitCode=0 Jul 28 13:42:58.002617 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:58.002493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerDied","Data":"a0300cda9a29d98e9bce3ac8781510184e7e34fe1686a626e3b8ae14374a8ee1"} Jul 28 13:42:59.776924 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:59.776889 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:42:59.776924 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:59.776923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:42:59.777466 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:42:59.776893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:42:59.777466 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:59.777011 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:42:59.777466 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:59.777077 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:42:59.777466 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:42:59.777132 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:43:00.008646 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:00.008615 2564 generic.go:356] "Generic (PLEG): container finished" podID="92cf0d6f-c2e8-4494-a431-1a339f34f906" containerID="6ecaaf92ea42bd8090b438694d0f48d4a569e899fcb7b0d32297a3626c834078" exitCode=0 Jul 28 13:43:00.008799 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:00.008662 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerDied","Data":"6ecaaf92ea42bd8090b438694d0f48d4a569e899fcb7b0d32297a3626c834078"} Jul 28 13:43:01.776878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:01.776709 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:43:01.776878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:01.776752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:43:01.776878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:01.776752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:43:01.776878 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:01.776852 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k4k9j" podUID="6f6e6450-cd8c-4979-90e5-06011a1d268c" Jul 28 13:43:01.777381 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:01.776932 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r6j7c" podUID="24616ef5-ef24-4816-a359-07511a2e6c75" Jul 28 13:43:01.777381 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:01.776998 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cmjgg" podUID="9d7a2dec-a85e-4a73-b3ad-245c59d03c26" Jul 28 13:43:02.468232 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:02.468188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:43:02.468388 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:02.468248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:43:02.468388 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.468363 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:43:02.468479 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.468400 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 13:43:02.468479 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.468437 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:43:34.468416574 +0000 UTC m=+66.169959895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 13:43:02.468479 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.468456 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret podName:9d7a2dec-a85e-4a73-b3ad-245c59d03c26 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:34.46844916 +0000 UTC m=+66.169992465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret") pod "global-pull-secret-syncer-cmjgg" (UID: "9d7a2dec-a85e-4a73-b3ad-245c59d03c26") : object "kube-system"/"original-pull-secret" not registered Jul 28 13:43:02.569491 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:02.569427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:43:02.569701 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.569649 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 13:43:02.569701 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.569676 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 13:43:02.569701 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.569689 2564 projected.go:196] Error preparing data for projected volume kube-api-access-549v4 for pod openshift-network-diagnostics/network-check-target-r6j7c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:43:02.569867 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:02.569768 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4 podName:24616ef5-ef24-4816-a359-07511a2e6c75 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:34.569747803 +0000 UTC m=+66.271291130 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-549v4" (UniqueName: "kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4") pod "network-check-target-r6j7c" (UID: "24616ef5-ef24-4816-a359-07511a2e6c75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 13:43:03.061047 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.061016 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-116.ec2.internal" event="NodeReady" Jul 28 13:43:03.061639 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.061188 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 13:43:03.126120 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.126088 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qsw4v"] Jul 28 13:43:03.131512 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.131488 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b2b7d"] Jul 28 13:43:03.131689 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.131649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.135220 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.135196 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.135220 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.135214 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8spsv\"" Jul 28 13:43:03.135630 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.135609 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 13:43:03.135746 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.135713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 13:43:03.140611 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.140594 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 13:43:03.140839 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.140820 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 13:43:03.140919 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.140883 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8rh7w\"" Jul 28 13:43:03.141589 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.141561 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 13:43:03.152076 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.152050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsw4v"] Jul 28 13:43:03.152350 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.152331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2b7d"] Jul 28 13:43:03.275478 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.275432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e40087-a96f-4656-8059-eae9600913e5-cert\") pod \"ingress-canary-b2b7d\" (UID: \"99e40087-a96f-4656-8059-eae9600913e5\") " pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.275690 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.275499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-tmp-dir\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.275690 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.275532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rfp\" (UniqueName: \"kubernetes.io/projected/99e40087-a96f-4656-8059-eae9600913e5-kube-api-access-p5rfp\") pod \"ingress-canary-b2b7d\" (UID: \"99e40087-a96f-4656-8059-eae9600913e5\") " pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.275690 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.275562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jm29\" (UniqueName: \"kubernetes.io/projected/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-kube-api-access-9jm29\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.275851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.275701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-config-volume\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.275851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.275744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.376815 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.376764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e40087-a96f-4656-8059-eae9600913e5-cert\") pod \"ingress-canary-b2b7d\" (UID: \"99e40087-a96f-4656-8059-eae9600913e5\") " pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.377007 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.376843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-tmp-dir\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.377007 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.376866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rfp\" (UniqueName: \"kubernetes.io/projected/99e40087-a96f-4656-8059-eae9600913e5-kube-api-access-p5rfp\") pod \"ingress-canary-b2b7d\" (UID: \"99e40087-a96f-4656-8059-eae9600913e5\") " pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.377007 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.376889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jm29\" (UniqueName: \"kubernetes.io/projected/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-kube-api-access-9jm29\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.377007 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.376958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-config-volume\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.377007 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.376985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.377237 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:03.377103 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 13:43:03.377237 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:03.377148 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls podName:2bc33d9d-f9d6-40b3-83ce-dc1aebf54246 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:03.877134884 +0000 UTC m=+35.578678190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls") pod "dns-default-qsw4v" (UID: "2bc33d9d-f9d6-40b3-83ce-dc1aebf54246") : secret "dns-default-metrics-tls" not found Jul 28 13:43:03.377237 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.377187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-tmp-dir\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.377697 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.377672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-config-volume\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.381832 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.381811 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e40087-a96f-4656-8059-eae9600913e5-cert\") pod \"ingress-canary-b2b7d\" (UID: \"99e40087-a96f-4656-8059-eae9600913e5\") " pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.387570 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.387546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jm29\" (UniqueName: \"kubernetes.io/projected/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-kube-api-access-9jm29\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.387718 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.387659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rfp\" (UniqueName: \"kubernetes.io/projected/99e40087-a96f-4656-8059-eae9600913e5-kube-api-access-p5rfp\") pod \"ingress-canary-b2b7d\" (UID: \"99e40087-a96f-4656-8059-eae9600913e5\") " pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.449645 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.449612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2b7d" Jul 28 13:43:03.777383 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.777344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:43:03.777383 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.777387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:43:03.777624 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.777344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:43:03.780532 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.780510 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 13:43:03.780532 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.780529 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 13:43:03.781296 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.781275 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kzm8z\"" Jul 28 13:43:03.781296 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.781288 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7bxcb\"" Jul 28 13:43:03.781449 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.781305 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 13:43:03.781449 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.781273 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 13:43:03.880717 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:03.880623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:03.880918 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:03.880746 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 13:43:03.880918 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:03.880817 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls podName:2bc33d9d-f9d6-40b3-83ce-dc1aebf54246 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:04.880796911 +0000 UTC m=+36.582340217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls") pod "dns-default-qsw4v" (UID: "2bc33d9d-f9d6-40b3-83ce-dc1aebf54246") : secret "dns-default-metrics-tls" not found Jul 28 13:43:04.242477 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:04.242281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2b7d"] Jul 28 13:43:04.249299 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:43:04.249274 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e40087_a96f_4656_8059_eae9600913e5.slice/crio-52595f528909c0cf99d1a14e0c85b1f8463f3c631c19fc404047e6d74db67d4b WatchSource:0}: Error finding container 52595f528909c0cf99d1a14e0c85b1f8463f3c631c19fc404047e6d74db67d4b: Status 404 returned error can't find the container with id 52595f528909c0cf99d1a14e0c85b1f8463f3c631c19fc404047e6d74db67d4b Jul 28 13:43:04.889046 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:04.889001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:04.889242 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:04.889147 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 13:43:04.889242 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:04.889207 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls podName:2bc33d9d-f9d6-40b3-83ce-dc1aebf54246 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:06.889193453 +0000 UTC m=+38.590736759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls") pod "dns-default-qsw4v" (UID: "2bc33d9d-f9d6-40b3-83ce-dc1aebf54246") : secret "dns-default-metrics-tls" not found Jul 28 13:43:05.022878 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:05.022847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2b7d" event={"ID":"99e40087-a96f-4656-8059-eae9600913e5","Type":"ContainerStarted","Data":"52595f528909c0cf99d1a14e0c85b1f8463f3c631c19fc404047e6d74db67d4b"} Jul 28 13:43:05.026058 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:05.026018 2564 generic.go:356] "Generic (PLEG): container finished" podID="92cf0d6f-c2e8-4494-a431-1a339f34f906" containerID="a25c2de7e3dd497772f959780a0b0f19471af4d5b57fb89e6bbd0d629a9f547b" exitCode=0 Jul 28 13:43:05.026187 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:05.026068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerDied","Data":"a25c2de7e3dd497772f959780a0b0f19471af4d5b57fb89e6bbd0d629a9f547b"} Jul 28 13:43:06.032012 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:06.031981 2564 generic.go:356] "Generic (PLEG): container finished" podID="92cf0d6f-c2e8-4494-a431-1a339f34f906" containerID="50e16491aed9f9f8eb47d37ccfb8c35d97cbea003abc107fd237545d5245616c" exitCode=0 Jul 28 13:43:06.032505 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:06.032054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerDied","Data":"50e16491aed9f9f8eb47d37ccfb8c35d97cbea003abc107fd237545d5245616c"} Jul 28 13:43:06.905568 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:06.905342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:06.905568 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:06.905494 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 13:43:06.905568 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:06.905566 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls podName:2bc33d9d-f9d6-40b3-83ce-dc1aebf54246 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:10.905550754 +0000 UTC m=+42.607094060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls") pod "dns-default-qsw4v" (UID: "2bc33d9d-f9d6-40b3-83ce-dc1aebf54246") : secret "dns-default-metrics-tls" not found Jul 28 13:43:07.035932 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:07.035897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2b7d" event={"ID":"99e40087-a96f-4656-8059-eae9600913e5","Type":"ContainerStarted","Data":"fa0c406ed3196e10fb64b9f1fff6cf5080fbf800d19ef27424eec20ae10d4edc"} Jul 28 13:43:07.038993 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:07.038971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" event={"ID":"92cf0d6f-c2e8-4494-a431-1a339f34f906","Type":"ContainerStarted","Data":"e540be25f025ff45f89efcc252c42555b0bf4dca1aca7dfeac78c6f0ad03ee04"} Jul 28 13:43:07.053335 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:07.053287 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b2b7d" podStartSLOduration=1.8192298789999999 podStartE2EDuration="4.053271264s" totalImagesPullingTime="2.234041385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:07.0531782 +0000 UTC m=+38.754721564" watchObservedRunningTime="2026-07-28 13:43:07.053271264 +0000 UTC m=+38.754814569" Jul 28 13:43:07.081248 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:07.081197 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rqqk5" podStartSLOduration=8.728132717 podStartE2EDuration="38.081183609s" totalImagesPullingTime="29.353050892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:07.079389409 +0000 UTC m=+38.780932755" watchObservedRunningTime="2026-07-28 13:43:07.081183609 +0000 UTC m=+38.782726937" Jul 28 13:43:10.934417 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:10.934378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:10.934855 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:10.934496 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 13:43:10.934855 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:10.934548 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls podName:2bc33d9d-f9d6-40b3-83ce-dc1aebf54246 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:18.934534963 +0000 UTC m=+50.636078270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls") pod "dns-default-qsw4v" (UID: "2bc33d9d-f9d6-40b3-83ce-dc1aebf54246") : secret "dns-default-metrics-tls" not found Jul 28 13:43:18.985855 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:18.985816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:18.986291 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:18.985958 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 13:43:18.986291 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:18.986025 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls podName:2bc33d9d-f9d6-40b3-83ce-dc1aebf54246 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:34.986009483 +0000 UTC m=+66.687552789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls") pod "dns-default-qsw4v" (UID: "2bc33d9d-f9d6-40b3-83ce-dc1aebf54246") : secret "dns-default-metrics-tls" not found Jul 28 13:43:29.025384 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:29.025357 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5cdj" Jul 28 13:43:34.491173 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.491119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:43:34.491748 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.491186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:43:34.494053 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.494033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 13:43:34.494252 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.494239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 13:43:34.502238 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:34.502216 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 13:43:34.502330 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:34.502282 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs podName:6f6e6450-cd8c-4979-90e5-06011a1d268c nodeName:}" failed. No retries permitted until 2026-07-28 13:44:38.502267168 +0000 UTC m=+130.203810475 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs") pod "network-metrics-daemon-k4k9j" (UID: "6f6e6450-cd8c-4979-90e5-06011a1d268c") : secret "metrics-daemon-secret" not found Jul 28 13:43:34.505933 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.505911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d7a2dec-a85e-4a73-b3ad-245c59d03c26-original-pull-secret\") pod \"global-pull-secret-syncer-cmjgg\" (UID: \"9d7a2dec-a85e-4a73-b3ad-245c59d03c26\") " pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:43:34.592074 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.592019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:43:34.595072 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.595049 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 13:43:34.604768 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.604744 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 13:43:34.614969 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.614945 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-549v4\" (UniqueName: \"kubernetes.io/projected/24616ef5-ef24-4816-a359-07511a2e6c75-kube-api-access-549v4\") pod \"network-check-target-r6j7c\" (UID: \"24616ef5-ef24-4816-a359-07511a2e6c75\") " pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:43:34.695571 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.695519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cmjgg" Jul 28 13:43:34.704403 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.704385 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kzm8z\"" Jul 28 13:43:34.711722 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.711702 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:43:34.821830 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.821800 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cmjgg"] Jul 28 13:43:34.824922 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:43:34.824875 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7a2dec_a85e_4a73_b3ad_245c59d03c26.slice/crio-7ed0f27d939cf5dad4ab99ecb8bd9a1fe4ca856c43a9c05fa170aedd1e944adc WatchSource:0}: Error finding container 7ed0f27d939cf5dad4ab99ecb8bd9a1fe4ca856c43a9c05fa170aedd1e944adc: Status 404 returned error can't find the container with id 7ed0f27d939cf5dad4ab99ecb8bd9a1fe4ca856c43a9c05fa170aedd1e944adc Jul 28 13:43:34.838549 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.838528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r6j7c"] Jul 28 13:43:34.841509 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:43:34.841487 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24616ef5_ef24_4816_a359_07511a2e6c75.slice/crio-9ffc6570904705d42b81adcc3ac81a6b967fabd704d8d81576dd68b3c694c668 WatchSource:0}: Error finding container 9ffc6570904705d42b81adcc3ac81a6b967fabd704d8d81576dd68b3c694c668: Status 404 returned error can't find the container with id 9ffc6570904705d42b81adcc3ac81a6b967fabd704d8d81576dd68b3c694c668 Jul 28 13:43:34.995142 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:34.995059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:43:34.995287 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:34.995199 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 13:43:34.995287 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:34.995259 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls podName:2bc33d9d-f9d6-40b3-83ce-dc1aebf54246 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:06.99524313 +0000 UTC m=+98.696786440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls") pod "dns-default-qsw4v" (UID: "2bc33d9d-f9d6-40b3-83ce-dc1aebf54246") : secret "dns-default-metrics-tls" not found Jul 28 13:43:35.099742 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:35.099711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6j7c" event={"ID":"24616ef5-ef24-4816-a359-07511a2e6c75","Type":"ContainerStarted","Data":"9ffc6570904705d42b81adcc3ac81a6b967fabd704d8d81576dd68b3c694c668"} Jul 28 13:43:35.100836 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:35.100805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cmjgg" event={"ID":"9d7a2dec-a85e-4a73-b3ad-245c59d03c26","Type":"ContainerStarted","Data":"7ed0f27d939cf5dad4ab99ecb8bd9a1fe4ca856c43a9c05fa170aedd1e944adc"} Jul 28 13:43:40.115655 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:40.115612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r6j7c" event={"ID":"24616ef5-ef24-4816-a359-07511a2e6c75","Type":"ContainerStarted","Data":"85bbbd3e491c8d26de9f2933c094d3ab582c7deaf8db3dd993479f1c03624c44"} Jul 28 13:43:40.116183 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:40.115774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:43:40.116973 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:40.116952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cmjgg" event={"ID":"9d7a2dec-a85e-4a73-b3ad-245c59d03c26","Type":"ContainerStarted","Data":"5a151e0392faf5301437a781935d0cc910462275345be2f131c4287eabe94bab"} Jul 28 13:43:40.134149 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:40.134107 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r6j7c" podStartSLOduration=66.575765336 podStartE2EDuration="1m11.134096182s" totalImagesPullingTime="4.558330846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:40.133885488 +0000 UTC m=+71.835428816" watchObservedRunningTime="2026-07-28 13:43:40.134096182 +0000 UTC m=+71.835639510" Jul 28 13:43:40.154712 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:40.154666 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cmjgg" podStartSLOduration=66.576133675 podStartE2EDuration="1m11.154654766s" totalImagesPullingTime="4.578521091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:40.153852852 +0000 UTC m=+71.855396192" watchObservedRunningTime="2026-07-28 13:43:40.154654766 +0000 UTC m=+71.856198094" Jul 28 13:43:48.953138 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.953099 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5"] Jul 28 13:43:48.956143 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.956124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:48.958791 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.958760 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 13:43:48.958791 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.958783 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 13:43:48.958959 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.958849 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 13:43:48.958959 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.958760 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 13:43:48.959676 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.959663 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-b6krg\"" Jul 28 13:43:48.965593 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.965559 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5"] Jul 28 13:43:48.982121 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.982087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd6104f8-624a-41ec-bda3-7ccba6de3105-config\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:48.982270 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.982154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd6104f8-624a-41ec-bda3-7ccba6de3105-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:48.982270 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:48.982244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htbjx\" (UniqueName: \"kubernetes.io/projected/bd6104f8-624a-41ec-bda3-7ccba6de3105-kube-api-access-htbjx\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.051503 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.051474 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6"] Jul 28 13:43:49.054261 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.054245 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:49.058760 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.058737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2n25b\"" Jul 28 13:43:49.058918 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.058897 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 28 13:43:49.058988 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.058921 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 28 13:43:49.069538 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.069515 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5464888c74-kr2h4"] Jul 28 13:43:49.072344 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.072325 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6"] Jul 28 13:43:49.072449 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.072434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.076839 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.076819 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 13:43:49.077001 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.076864 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 13:43:49.077155 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.077140 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j9bk9\"" Jul 28 13:43:49.077441 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.077426 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 13:43:49.082695 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-installation-pull-secrets\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.082800 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htbjx\" (UniqueName: \"kubernetes.io/projected/bd6104f8-624a-41ec-bda3-7ccba6de3105-kube-api-access-htbjx\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.082800 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24274456-35d5-411f-a0a3-bad2157ea5a6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:49.082800 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7d72\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-kube-api-access-w7d72\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.082800 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-certificates\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.083005 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.083005 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-trusted-ca\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.083005 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd6104f8-624a-41ec-bda3-7ccba6de3105-config\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.083005 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd6104f8-624a-41ec-bda3-7ccba6de3105-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.083005 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.082987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:49.083256 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.083010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-image-registry-private-configuration\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.083256 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.083029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea000a28-6cc0-4fc0-9a95-6e005069082e-ca-trust-extracted\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.083256 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.083127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-bound-sa-token\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.083402 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.083367 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bd6104f8-624a-41ec-bda3-7ccba6de3105-config\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.085136 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.085115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bd6104f8-624a-41ec-bda3-7ccba6de3105-serving-cert\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.085213 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.085140 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 13:43:49.092132 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.092109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5464888c74-kr2h4"] Jul 28 13:43:49.110702 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.110670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htbjx\" (UniqueName: \"kubernetes.io/projected/bd6104f8-624a-41ec-bda3-7ccba6de3105-kube-api-access-htbjx\") pod \"service-ca-operator-5d6c75f84f-zmts5\" (UID: \"bd6104f8-624a-41ec-bda3-7ccba6de3105\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.150269 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.150239 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj"] Jul 28 13:43:49.153234 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.153208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:49.154241 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.154216 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s"] Jul 28 13:43:49.156788 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.156729 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.157097 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.157075 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 28 13:43:49.157259 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.157241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 28 13:43:49.157319 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.157274 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7zp57\"" Jul 28 13:43:49.157504 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.157492 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 28 13:43:49.161207 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.161181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 13:43:49.162640 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.162618 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 13:43:49.162738 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.162672 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m8sk9\"" Jul 28 13:43:49.163260 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.163223 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 13:43:49.163874 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.163848 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 13:43:49.167140 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.167117 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj"] Jul 28 13:43:49.175431 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.175408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s"] Jul 28 13:43:49.183692 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:49.183791 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58s7\" (UniqueName: \"kubernetes.io/projected/8eeef8af-7631-4b9b-9cc1-09c168cff686-kube-api-access-t58s7\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:49.183791 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-image-registry-private-configuration\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.183791 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea000a28-6cc0-4fc0-9a95-6e005069082e-ca-trust-extracted\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.183791 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-bound-sa-token\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.183980 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.183804 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 13:43:49.183980 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.183864 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert podName:24274456-35d5-411f-a0a3-bad2157ea5a6 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:49.683847791 +0000 UTC m=+81.385391098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sg7c6" (UID: "24274456-35d5-411f-a0a3-bad2157ea5a6") : secret "networking-console-plugin-cert" not found Jul 28 13:43:49.183980 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:49.183980 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-installation-pull-secrets\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.183980 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24274456-35d5-411f-a0a3-bad2157ea5a6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:49.183980 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7d72\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-kube-api-access-w7d72\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.184283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.183991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dcc88e5c-0cde-4f78-9d31-34c61b544946-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.184283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvlpq\" (UniqueName: \"kubernetes.io/projected/dcc88e5c-0cde-4f78-9d31-34c61b544946-kube-api-access-gvlpq\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.184283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-certificates\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.184283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea000a28-6cc0-4fc0-9a95-6e005069082e-ca-trust-extracted\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.184283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.184283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.184283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-trusted-ca\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.184645 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.184333 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 13:43:49.184645 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.184346 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5464888c74-kr2h4: secret "image-registry-tls" not found Jul 28 13:43:49.184645 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.184410 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls podName:ea000a28-6cc0-4fc0-9a95-6e005069082e nodeName:}" failed. No retries permitted until 2026-07-28 13:43:49.684389945 +0000 UTC m=+81.385933253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls") pod "image-registry-5464888c74-kr2h4" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e") : secret "image-registry-tls" not found Jul 28 13:43:49.184803 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184755 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24274456-35d5-411f-a0a3-bad2157ea5a6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:49.184803 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.184785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-certificates\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.185426 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.185400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-trusted-ca\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.186342 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.186324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-installation-pull-secrets\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.186421 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.186369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-image-registry-private-configuration\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.194033 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.194015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-bound-sa-token\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.197382 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.197361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7d72\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-kube-api-access-w7d72\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.264965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.264936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" Jul 28 13:43:49.285037 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.285010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.285169 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.285085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58s7\" (UniqueName: \"kubernetes.io/projected/8eeef8af-7631-4b9b-9cc1-09c168cff686-kube-api-access-t58s7\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:49.285169 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.285111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:49.285169 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.285148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dcc88e5c-0cde-4f78-9d31-34c61b544946-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.285169 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.285164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvlpq\" (UniqueName: \"kubernetes.io/projected/dcc88e5c-0cde-4f78-9d31-34c61b544946-kube-api-access-gvlpq\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.285370 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.285181 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:49.285370 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.285267 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls podName:dcc88e5c-0cde-4f78-9d31-34c61b544946 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:49.785247015 +0000 UTC m=+81.486790329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j6d4s" (UID: "dcc88e5c-0cde-4f78-9d31-34c61b544946") : secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:49.285370 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.285270 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 13:43:49.285370 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.285332 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls podName:8eeef8af-7631-4b9b-9cc1-09c168cff686 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:49.785315781 +0000 UTC m=+81.486859101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls") pod "cluster-samples-operator-6748489656-2ffbj" (UID: "8eeef8af-7631-4b9b-9cc1-09c168cff686") : secret "samples-operator-tls" not found Jul 28 13:43:49.285901 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.285881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dcc88e5c-0cde-4f78-9d31-34c61b544946-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.296039 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.296018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvlpq\" (UniqueName: \"kubernetes.io/projected/dcc88e5c-0cde-4f78-9d31-34c61b544946-kube-api-access-gvlpq\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.296141 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.296092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58s7\" (UniqueName: \"kubernetes.io/projected/8eeef8af-7631-4b9b-9cc1-09c168cff686-kube-api-access-t58s7\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:49.384674 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.384638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5"] Jul 28 13:43:49.387900 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:43:49.387870 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6104f8_624a_41ec_bda3_7ccba6de3105.slice/crio-442dc321e6a1a76f64dcac7e2c97abf48d6612c7bcf9ed24d83dded74879fd4b WatchSource:0}: Error finding container 442dc321e6a1a76f64dcac7e2c97abf48d6612c7bcf9ed24d83dded74879fd4b: Status 404 returned error can't find the container with id 442dc321e6a1a76f64dcac7e2c97abf48d6612c7bcf9ed24d83dded74879fd4b Jul 28 13:43:49.687237 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.687150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:49.687237 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.687215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:49.687427 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.687310 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 13:43:49.687427 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.687325 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5464888c74-kr2h4: secret "image-registry-tls" not found Jul 28 13:43:49.687427 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.687369 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 13:43:49.687427 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.687376 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls podName:ea000a28-6cc0-4fc0-9a95-6e005069082e nodeName:}" failed. No retries permitted until 2026-07-28 13:43:50.687362315 +0000 UTC m=+82.388905624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls") pod "image-registry-5464888c74-kr2h4" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e") : secret "image-registry-tls" not found Jul 28 13:43:49.687552 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.687434 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert podName:24274456-35d5-411f-a0a3-bad2157ea5a6 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:50.687420786 +0000 UTC m=+82.388964093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sg7c6" (UID: "24274456-35d5-411f-a0a3-bad2157ea5a6") : secret "networking-console-plugin-cert" not found Jul 28 13:43:49.788491 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.788453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:49.788679 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:49.788530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:49.788679 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.788608 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:49.788679 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.788651 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 13:43:49.788679 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.788674 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls podName:dcc88e5c-0cde-4f78-9d31-34c61b544946 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:50.7886583 +0000 UTC m=+82.490201606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j6d4s" (UID: "dcc88e5c-0cde-4f78-9d31-34c61b544946") : secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:49.788827 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:49.788690 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls podName:8eeef8af-7631-4b9b-9cc1-09c168cff686 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:50.788679749 +0000 UTC m=+82.490223054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls") pod "cluster-samples-operator-6748489656-2ffbj" (UID: "8eeef8af-7631-4b9b-9cc1-09c168cff686") : secret "samples-operator-tls" not found Jul 28 13:43:50.141653 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:50.141612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" event={"ID":"bd6104f8-624a-41ec-bda3-7ccba6de3105","Type":"ContainerStarted","Data":"442dc321e6a1a76f64dcac7e2c97abf48d6612c7bcf9ed24d83dded74879fd4b"} Jul 28 13:43:50.694672 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:50.694639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:50.694846 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:50.694711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:50.694846 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.694780 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 13:43:50.694846 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.694801 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5464888c74-kr2h4: secret "image-registry-tls" not found Jul 28 13:43:50.694846 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.694841 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 13:43:50.694977 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.694858 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls podName:ea000a28-6cc0-4fc0-9a95-6e005069082e nodeName:}" failed. No retries permitted until 2026-07-28 13:43:52.694839347 +0000 UTC m=+84.396382675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls") pod "image-registry-5464888c74-kr2h4" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e") : secret "image-registry-tls" not found Jul 28 13:43:50.694977 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.694886 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert podName:24274456-35d5-411f-a0a3-bad2157ea5a6 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:52.694874752 +0000 UTC m=+84.396418058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sg7c6" (UID: "24274456-35d5-411f-a0a3-bad2157ea5a6") : secret "networking-console-plugin-cert" not found Jul 28 13:43:50.796103 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:50.796067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:50.796292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:50.796228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:50.796292 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.796232 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:50.796411 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.796304 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls podName:dcc88e5c-0cde-4f78-9d31-34c61b544946 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:52.796285248 +0000 UTC m=+84.497828554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j6d4s" (UID: "dcc88e5c-0cde-4f78-9d31-34c61b544946") : secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:50.796411 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.796350 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 13:43:50.796411 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:50.796401 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls podName:8eeef8af-7631-4b9b-9cc1-09c168cff686 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:52.796385 +0000 UTC m=+84.497928309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls") pod "cluster-samples-operator-6748489656-2ffbj" (UID: "8eeef8af-7631-4b9b-9cc1-09c168cff686") : secret "samples-operator-tls" not found Jul 28 13:43:52.147456 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.147415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" event={"ID":"bd6104f8-624a-41ec-bda3-7ccba6de3105","Type":"ContainerStarted","Data":"00a318479f11b2d993f7549d3d42e2199db97d9d9168a3dfa6b36d68b334b83b"} Jul 28 13:43:52.168258 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.168201 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" podStartSLOduration=2.082155761 podStartE2EDuration="4.168183605s" totalImagesPullingTime="2.086027844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:52.167328774 +0000 UTC m=+83.868872104" watchObservedRunningTime="2026-07-28 13:43:52.168183605 +0000 UTC m=+83.869726910" Jul 28 13:43:52.714365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.714327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:52.714574 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.714389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:52.714574 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.714508 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 13:43:52.714574 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.714535 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5464888c74-kr2h4: secret "image-registry-tls" not found Jul 28 13:43:52.714574 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.714544 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 13:43:52.714574 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.714624 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls podName:ea000a28-6cc0-4fc0-9a95-6e005069082e nodeName:}" failed. No retries permitted until 2026-07-28 13:43:56.714604303 +0000 UTC m=+88.416147611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls") pod "image-registry-5464888c74-kr2h4" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e") : secret "image-registry-tls" not found Jul 28 13:43:52.714827 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.714648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert podName:24274456-35d5-411f-a0a3-bad2157ea5a6 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:56.714635831 +0000 UTC m=+88.416179138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sg7c6" (UID: "24274456-35d5-411f-a0a3-bad2157ea5a6") : secret "networking-console-plugin-cert" not found Jul 28 13:43:52.762352 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.762313 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lxwmm"] Jul 28 13:43:52.765731 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.765711 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.769694 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.769658 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 13:43:52.770823 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.770795 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 13:43:52.770823 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.770817 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 13:43:52.771011 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.770796 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z2kzl\"" Jul 28 13:43:52.771011 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.770797 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 13:43:52.787046 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.787017 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lxwmm"] Jul 28 13:43:52.815223 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.815191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.815461 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.815234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/660b57a2-689c-49b9-a48d-5bebc4138ff5-crio-socket\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.815461 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.815260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/660b57a2-689c-49b9-a48d-5bebc4138ff5-data-volume\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.815461 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.815288 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.815461 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.815319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:52.815461 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.815359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlhf\" (UniqueName: \"kubernetes.io/projected/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-api-access-ddlhf\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.815461 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.815408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:52.815461 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.815453 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 13:43:52.815731 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.815501 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:52.815731 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.815519 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls podName:8eeef8af-7631-4b9b-9cc1-09c168cff686 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:56.815502159 +0000 UTC m=+88.517045465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls") pod "cluster-samples-operator-6748489656-2ffbj" (UID: "8eeef8af-7631-4b9b-9cc1-09c168cff686") : secret "samples-operator-tls" not found Jul 28 13:43:52.815731 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.815540 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls podName:dcc88e5c-0cde-4f78-9d31-34c61b544946 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:56.815528433 +0000 UTC m=+88.517071739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j6d4s" (UID: "dcc88e5c-0cde-4f78-9d31-34c61b544946") : secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:52.916240 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.916205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.916389 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.916269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlhf\" (UniqueName: \"kubernetes.io/projected/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-api-access-ddlhf\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.916389 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.916323 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 13:43:52.916477 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.916403 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 13:43:52.916477 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.916334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.916477 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.916407 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:53.416386097 +0000 UTC m=+85.117929421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : configmap "kube-rbac-proxy" not found Jul 28 13:43:52.916477 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:52.916471 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:53.416449453 +0000 UTC m=+85.117992771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : secret "insights-runtime-extractor-tls" not found Jul 28 13:43:52.916636 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.916510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/660b57a2-689c-49b9-a48d-5bebc4138ff5-crio-socket\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.916636 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.916546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/660b57a2-689c-49b9-a48d-5bebc4138ff5-data-volume\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.916636 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.916566 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/660b57a2-689c-49b9-a48d-5bebc4138ff5-crio-socket\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.916874 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.916857 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/660b57a2-689c-49b9-a48d-5bebc4138ff5-data-volume\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:52.928019 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:52.927986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlhf\" (UniqueName: \"kubernetes.io/projected/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-api-access-ddlhf\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:53.421255 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:53.421215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:53.421661 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:53.421267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:53.421661 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:53.421339 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 13:43:53.421661 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:53.421370 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 13:43:53.421661 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:53.421396 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:54.421380471 +0000 UTC m=+86.122923778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : configmap "kube-rbac-proxy" not found Jul 28 13:43:53.421661 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:53.421425 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:54.421411121 +0000 UTC m=+86.122954428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : secret "insights-runtime-extractor-tls" not found Jul 28 13:43:54.430086 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:54.430044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:54.430514 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:54.430097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:54.430514 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:54.430160 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 13:43:54.430514 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:54.430189 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 13:43:54.430514 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:54.430208 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:56.430194077 +0000 UTC m=+88.131737383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : configmap "kube-rbac-proxy" not found Jul 28 13:43:54.430514 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:54.430242 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:43:56.430229464 +0000 UTC m=+88.131772769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : secret "insights-runtime-extractor-tls" not found Jul 28 13:43:54.523150 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:54.523124 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wvn7w_550041e0-5515-4eeb-98b2-23273b7c848d/dns-node-resolver/0.log" Jul 28 13:43:55.919483 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:55.919455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tz84d_13431ed3-6ada-4d5d-b763-b30d0fd9c039/node-ca/0.log" Jul 28 13:43:56.448005 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:56.447959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:56.448190 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.448080 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 13:43:56.448190 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:56.448107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:43:56.448190 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.448145 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:00.448127757 +0000 UTC m=+92.149671081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : configmap "kube-rbac-proxy" not found Jul 28 13:43:56.448324 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.448233 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 13:43:56.448324 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.448278 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:00.448266657 +0000 UTC m=+92.149809966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : secret "insights-runtime-extractor-tls" not found Jul 28 13:43:56.518054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:56.518023 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2b7d_99e40087-a96f-4656-8059-eae9600913e5/serve-healthcheck-canary/0.log" Jul 28 13:43:56.750876 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:56.750836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:43:56.751034 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:56.750890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:43:56.751034 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.750995 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 13:43:56.751034 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.751010 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 13:43:56.751034 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.751023 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5464888c74-kr2h4: secret "image-registry-tls" not found Jul 28 13:43:56.751155 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.751061 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert podName:24274456-35d5-411f-a0a3-bad2157ea5a6 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:04.751045461 +0000 UTC m=+96.452588767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sg7c6" (UID: "24274456-35d5-411f-a0a3-bad2157ea5a6") : secret "networking-console-plugin-cert" not found Jul 28 13:43:56.751155 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.751075 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls podName:ea000a28-6cc0-4fc0-9a95-6e005069082e nodeName:}" failed. No retries permitted until 2026-07-28 13:44:04.751068942 +0000 UTC m=+96.452612247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls") pod "image-registry-5464888c74-kr2h4" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e") : secret "image-registry-tls" not found Jul 28 13:43:56.852041 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:56.851990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:43:56.852236 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:43:56.852092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:43:56.852236 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.852176 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 13:43:56.852387 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.852260 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 13:43:56.852387 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.852278 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls podName:8eeef8af-7631-4b9b-9cc1-09c168cff686 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:04.852258705 +0000 UTC m=+96.553802031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls") pod "cluster-samples-operator-6748489656-2ffbj" (UID: "8eeef8af-7631-4b9b-9cc1-09c168cff686") : secret "samples-operator-tls" not found Jul 28 13:43:56.852387 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:43:56.852316 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls podName:dcc88e5c-0cde-4f78-9d31-34c61b544946 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:04.852301369 +0000 UTC m=+96.553844675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j6d4s" (UID: "dcc88e5c-0cde-4f78-9d31-34c61b544946") : secret "cluster-monitoring-operator-tls" not found Jul 28 13:44:00.479961 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:00.479922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:00.480374 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:00.480049 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 13:44:00.480374 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:00.480107 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:08.480093117 +0000 UTC m=+100.181636440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : configmap "kube-rbac-proxy" not found Jul 28 13:44:00.480374 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:00.480174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:00.480374 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:00.480291 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 13:44:00.480374 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:00.480329 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:08.480317104 +0000 UTC m=+100.181860430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : secret "insights-runtime-extractor-tls" not found Jul 28 13:44:04.817425 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:04.817383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:44:04.817425 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:04.817445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:44:04.817891 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:04.817532 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 28 13:44:04.817891 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:04.817649 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert podName:24274456-35d5-411f-a0a3-bad2157ea5a6 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:20.817630789 +0000 UTC m=+112.519174094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-sg7c6" (UID: "24274456-35d5-411f-a0a3-bad2157ea5a6") : secret "networking-console-plugin-cert" not found Jul 28 13:44:04.819790 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:04.819770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"image-registry-5464888c74-kr2h4\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:44:04.918854 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:04.918813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:44:04.919021 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:04.918884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:44:04.919021 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:04.918965 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 13:44:04.919102 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:04.919034 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls podName:dcc88e5c-0cde-4f78-9d31-34c61b544946 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:20.919019355 +0000 UTC m=+112.620562661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j6d4s" (UID: "dcc88e5c-0cde-4f78-9d31-34c61b544946") : secret "cluster-monitoring-operator-tls" not found Jul 28 13:44:04.921320 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:04.921296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8eeef8af-7631-4b9b-9cc1-09c168cff686-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2ffbj\" (UID: \"8eeef8af-7631-4b9b-9cc1-09c168cff686\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:44:04.980473 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:04.980431 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:44:05.064871 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:05.064840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" Jul 28 13:44:05.107688 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:05.107664 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5464888c74-kr2h4"] Jul 28 13:44:05.109796 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:05.109770 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea000a28_6cc0_4fc0_9a95_6e005069082e.slice/crio-f28eb14419c40d25134ebf89b950cc19128c46bd692d32e92bcaf24742ca1f30 WatchSource:0}: Error finding container f28eb14419c40d25134ebf89b950cc19128c46bd692d32e92bcaf24742ca1f30: Status 404 returned error can't find the container with id f28eb14419c40d25134ebf89b950cc19128c46bd692d32e92bcaf24742ca1f30 Jul 28 13:44:05.179879 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:05.179846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" event={"ID":"ea000a28-6cc0-4fc0-9a95-6e005069082e","Type":"ContainerStarted","Data":"b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60"} Jul 28 13:44:05.180014 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:05.179906 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" event={"ID":"ea000a28-6cc0-4fc0-9a95-6e005069082e","Type":"ContainerStarted","Data":"f28eb14419c40d25134ebf89b950cc19128c46bd692d32e92bcaf24742ca1f30"} Jul 28 13:44:05.180014 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:05.179980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:44:05.194147 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:05.194120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj"] Jul 28 13:44:05.202149 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:05.202099 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" podStartSLOduration=16.202084674 podStartE2EDuration="16.202084674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:05.201109623 +0000 UTC m=+96.902652965" watchObservedRunningTime="2026-07-28 13:44:05.202084674 +0000 UTC m=+96.903628000" Jul 28 13:44:06.183990 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:06.183930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" event={"ID":"8eeef8af-7631-4b9b-9cc1-09c168cff686","Type":"ContainerStarted","Data":"4aea7d2a8f11787121df41ea45f1ac68bde03da7a68d141dff4f564266c2995d"} Jul 28 13:44:07.037494 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.037469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:44:07.039924 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.039869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc33d9d-f9d6-40b3-83ce-dc1aebf54246-metrics-tls\") pod \"dns-default-qsw4v\" (UID: \"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246\") " pod="openshift-dns/dns-default-qsw4v" Jul 28 13:44:07.046833 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.046815 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8spsv\"" Jul 28 13:44:07.054453 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.054432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsw4v" Jul 28 13:44:07.184924 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.184836 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsw4v"] Jul 28 13:44:07.189340 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:07.189306 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc33d9d_f9d6_40b3_83ce_dc1aebf54246.slice/crio-95537c37e40b55c69bd6fa2b6281f41dc63e9053206e665a8b1ad1227c647b3f WatchSource:0}: Error finding container 95537c37e40b55c69bd6fa2b6281f41dc63e9053206e665a8b1ad1227c647b3f: Status 404 returned error can't find the container with id 95537c37e40b55c69bd6fa2b6281f41dc63e9053206e665a8b1ad1227c647b3f Jul 28 13:44:07.189704 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.189679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" event={"ID":"8eeef8af-7631-4b9b-9cc1-09c168cff686","Type":"ContainerStarted","Data":"67c97b303ad8bea397363477bf477a6fd9c838c9f60add92a600318c6d6379d3"} Jul 28 13:44:07.189769 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.189733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" event={"ID":"8eeef8af-7631-4b9b-9cc1-09c168cff686","Type":"ContainerStarted","Data":"834f0d0ae2e217c3394ab03d6b526097f6634b15a825383cc392f4875cf0a7e4"} Jul 28 13:44:07.212037 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:07.211989 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2ffbj" podStartSLOduration=16.524334254 podStartE2EDuration="18.211970368s" totalImagesPullingTime="1.687636114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:07.210361696 +0000 UTC m=+98.911905024" watchObservedRunningTime="2026-07-28 13:44:07.211970368 +0000 UTC m=+98.913513695" Jul 28 13:44:08.195338 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:08.195268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsw4v" event={"ID":"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246","Type":"ContainerStarted","Data":"95537c37e40b55c69bd6fa2b6281f41dc63e9053206e665a8b1ad1227c647b3f"} Jul 28 13:44:08.551368 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:08.551318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:08.551523 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:08.551478 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 13:44:08.551572 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:08.551561 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls podName:660b57a2-689c-49b9-a48d-5bebc4138ff5 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:24.551545282 +0000 UTC m=+116.253088591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxwmm" (UID: "660b57a2-689c-49b9-a48d-5bebc4138ff5") : secret "insights-runtime-extractor-tls" not found Jul 28 13:44:08.551638 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:08.551475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:08.551993 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:08.551975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/660b57a2-689c-49b9-a48d-5bebc4138ff5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:09.199203 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:09.199168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsw4v" event={"ID":"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246","Type":"ContainerStarted","Data":"dadd26fac8de02dd34df1201448b3132c6902ba6ec97aee8c5895e089bc908a5"} Jul 28 13:44:09.199649 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:09.199220 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsw4v" event={"ID":"2bc33d9d-f9d6-40b3-83ce-dc1aebf54246","Type":"ContainerStarted","Data":"4c6027e73c9390457e2ec57f552c41b8d2b436253fda02c385131556fc355fe3"} Jul 28 13:44:09.199649 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:09.199259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qsw4v" Jul 28 13:44:09.222827 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:09.222765 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qsw4v" podStartSLOduration=64.858434309 podStartE2EDuration="1m6.222746405s" totalImagesPullingTime="1.364312096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:09.221204734 +0000 UTC m=+100.922748062" watchObservedRunningTime="2026-07-28 13:44:09.222746405 +0000 UTC m=+100.924289732" Jul 28 13:44:11.122461 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:11.122431 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r6j7c" Jul 28 13:44:18.450291 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.450257 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6"] Jul 28 13:44:18.454160 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.454140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:18.459309 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:18.459282 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Jul 28 13:44:18.460501 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:18.460461 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" podUID="55b911d9-835c-4940-82eb-bb094edcf30b" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:18.460664 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:18.460642 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" Jul 28 13:44:18.460788 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:18.460769 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-dockercfg-q2nw4\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q2nw4\"" type="*v1.Secret" Jul 28 13:44:18.461116 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:18.461096 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 28 13:44:18.461242 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:18.461224 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 28 13:44:18.482631 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.482602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6"] Jul 28 13:44:18.535036 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.535007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847xk\" (UniqueName: \"kubernetes.io/projected/55b911d9-835c-4940-82eb-bb094edcf30b-kube-api-access-847xk\") pod \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" (UID: \"55b911d9-835c-4940-82eb-bb094edcf30b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:18.535207 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.535059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55b911d9-835c-4940-82eb-bb094edcf30b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" (UID: \"55b911d9-835c-4940-82eb-bb094edcf30b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:18.564599 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.562937 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5464888c74-kr2h4"] Jul 28 13:44:18.610448 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.610415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-859d744f44-476ss"] Jul 28 13:44:18.612947 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.612930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.635827 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.635797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847xk\" (UniqueName: \"kubernetes.io/projected/55b911d9-835c-4940-82eb-bb094edcf30b-kube-api-access-847xk\") pod \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" (UID: \"55b911d9-835c-4940-82eb-bb094edcf30b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:18.636008 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.635851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55b911d9-835c-4940-82eb-bb094edcf30b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" (UID: \"55b911d9-835c-4940-82eb-bb094edcf30b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:18.637161 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.637136 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859d744f44-476ss"] Jul 28 13:44:18.737012 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.736977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-registry-tls\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.737177 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.737022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-installation-pull-secrets\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.737177 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.737124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-bound-sa-token\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.737177 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.737148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-image-registry-private-configuration\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.737177 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.737172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-ca-trust-extracted\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.737345 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.737245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw796\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-kube-api-access-fw796\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.737345 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.737289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-trusted-ca\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.737345 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.737321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-registry-certificates\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.837774 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-bound-sa-token\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.837774 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-image-registry-private-configuration\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-ca-trust-extracted\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw796\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-kube-api-access-fw796\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-trusted-ca\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-registry-certificates\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-registry-tls\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.837973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-installation-pull-secrets\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838288 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.838263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-ca-trust-extracted\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.838930 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.838907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-registry-certificates\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.839054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.838937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-trusted-ca\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.840311 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.840282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-image-registry-private-configuration\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.840409 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.840329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-installation-pull-secrets\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.840641 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.840620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-registry-tls\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.851876 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.851854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw796\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-kube-api-access-fw796\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.852168 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.852153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25-bound-sa-token\") pod \"image-registry-859d744f44-476ss\" (UID: \"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25\") " pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:18.921173 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:18.921135 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:19.057863 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.057833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859d744f44-476ss"] Jul 28 13:44:19.061784 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:19.061760 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fe3421_0e3b_4a57_8648_8bfe1a0d7d25.slice/crio-e1bcd06ea3e85c63d121c4ae48f02e7a116eb0e3b901e802e727112577a373b4 WatchSource:0}: Error finding container e1bcd06ea3e85c63d121c4ae48f02e7a116eb0e3b901e802e727112577a373b4: Status 404 returned error can't find the container with id e1bcd06ea3e85c63d121c4ae48f02e7a116eb0e3b901e802e727112577a373b4 Jul 28 13:44:19.204406 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.204378 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qsw4v" Jul 28 13:44:19.231415 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.231376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859d744f44-476ss" event={"ID":"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25","Type":"ContainerStarted","Data":"fab539403e78033ffe1548d7150c20d3c130df75a373ee24e72baf43b4c84aed"} Jul 28 13:44:19.231621 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.231430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859d744f44-476ss" event={"ID":"f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25","Type":"ContainerStarted","Data":"e1bcd06ea3e85c63d121c4ae48f02e7a116eb0e3b901e802e727112577a373b4"} Jul 28 13:44:19.231621 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.231480 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:19.258746 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.258621 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-859d744f44-476ss" podStartSLOduration=1.258594956 podStartE2EDuration="1.258594956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:19.256915632 +0000 UTC m=+110.958458960" watchObservedRunningTime="2026-07-28 13:44:19.258594956 +0000 UTC m=+110.960138280" Jul 28 13:44:19.317368 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.317332 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 28 13:44:19.329296 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.329257 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55b911d9-835c-4940-82eb-bb094edcf30b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" (UID: \"55b911d9-835c-4940-82eb-bb094edcf30b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:19.576719 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.576634 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 13:44:19.647163 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:19.647126 2564 projected.go:291] Couldn't get configMap open-cluster-management-agent-addon/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 28 13:44:19.647276 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:19.647173 2564 projected.go:196] Error preparing data for projected volume kube-api-access-847xk for pod open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6: failed to sync configmap cache: timed out waiting for the condition Jul 28 13:44:19.647276 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:19.647232 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55b911d9-835c-4940-82eb-bb094edcf30b-kube-api-access-847xk podName:55b911d9-835c-4940-82eb-bb094edcf30b nodeName:}" failed. No retries permitted until 2026-07-28 13:44:20.147212914 +0000 UTC m=+111.848756219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-847xk" (UniqueName: "kubernetes.io/projected/55b911d9-835c-4940-82eb-bb094edcf30b-kube-api-access-847xk") pod "managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" (UID: "55b911d9-835c-4940-82eb-bb094edcf30b") : failed to sync configmap cache: timed out waiting for the condition Jul 28 13:44:19.820160 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.820129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q2nw4\"" Jul 28 13:44:19.897745 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.897646 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 13:44:19.920702 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:19.920676 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 13:44:20.149956 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.149857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847xk\" (UniqueName: \"kubernetes.io/projected/55b911d9-835c-4940-82eb-bb094edcf30b-kube-api-access-847xk\") pod \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" (UID: \"55b911d9-835c-4940-82eb-bb094edcf30b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:20.152529 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.152494 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-847xk\" (UniqueName: \"kubernetes.io/projected/55b911d9-835c-4940-82eb-bb094edcf30b-kube-api-access-847xk\") pod \"managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6\" (UID: \"55b911d9-835c-4940-82eb-bb094edcf30b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:20.273475 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.273423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" Jul 28 13:44:20.401207 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.401127 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6"] Jul 28 13:44:20.404063 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:20.404037 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55b911d9_835c_4940_82eb_bb094edcf30b.slice/crio-4b9694ba74c5e8de2cb895d3785ce5e6107640d2c204ce13c7d4063d05754356 WatchSource:0}: Error finding container 4b9694ba74c5e8de2cb895d3785ce5e6107640d2c204ce13c7d4063d05754356: Status 404 returned error can't find the container with id 4b9694ba74c5e8de2cb895d3785ce5e6107640d2c204ce13c7d4063d05754356 Jul 28 13:44:20.857023 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.856988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:44:20.859438 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.859412 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24274456-35d5-411f-a0a3-bad2157ea5a6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-sg7c6\" (UID: \"24274456-35d5-411f-a0a3-bad2157ea5a6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:44:20.863300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.863277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" Jul 28 13:44:20.958473 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.958437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:44:20.961387 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.961356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dcc88e5c-0cde-4f78-9d31-34c61b544946-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j6d4s\" (UID: \"dcc88e5c-0cde-4f78-9d31-34c61b544946\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:44:20.971069 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:20.971041 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" Jul 28 13:44:21.013109 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:21.013009 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6"] Jul 28 13:44:21.015485 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:21.015457 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24274456_35d5_411f_a0a3_bad2157ea5a6.slice/crio-0158ad6505fbf299b460fc829a3be138ac3544c661be803ea7b2e8c70bfd1d13 WatchSource:0}: Error finding container 0158ad6505fbf299b460fc829a3be138ac3544c661be803ea7b2e8c70bfd1d13: Status 404 returned error can't find the container with id 0158ad6505fbf299b460fc829a3be138ac3544c661be803ea7b2e8c70bfd1d13 Jul 28 13:44:21.099667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:21.099634 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s"] Jul 28 13:44:21.102702 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:21.102673 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc88e5c_0cde_4f78_9d31_34c61b544946.slice/crio-fd1414be8c2f2aaad5139c0b9cd6c206e181c17b66ca120025b4771b150af79c WatchSource:0}: Error finding container fd1414be8c2f2aaad5139c0b9cd6c206e181c17b66ca120025b4771b150af79c: Status 404 returned error can't find the container with id fd1414be8c2f2aaad5139c0b9cd6c206e181c17b66ca120025b4771b150af79c Jul 28 13:44:21.239992 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:21.239953 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" event={"ID":"55b911d9-835c-4940-82eb-bb094edcf30b","Type":"ContainerStarted","Data":"4b9694ba74c5e8de2cb895d3785ce5e6107640d2c204ce13c7d4063d05754356"} Jul 28 13:44:21.241125 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:21.241093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" event={"ID":"24274456-35d5-411f-a0a3-bad2157ea5a6","Type":"ContainerStarted","Data":"0158ad6505fbf299b460fc829a3be138ac3544c661be803ea7b2e8c70bfd1d13"} Jul 28 13:44:21.242075 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:21.242057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" event={"ID":"dcc88e5c-0cde-4f78-9d31-34c61b544946","Type":"ContainerStarted","Data":"fd1414be8c2f2aaad5139c0b9cd6c206e181c17b66ca120025b4771b150af79c"} Jul 28 13:44:24.253981 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.253944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" event={"ID":"dcc88e5c-0cde-4f78-9d31-34c61b544946","Type":"ContainerStarted","Data":"fc0275951c354b95c8ecae73782a88405286a9643e9c7244273a8470e92e2dbe"} Jul 28 13:44:24.255415 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.255388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" event={"ID":"55b911d9-835c-4940-82eb-bb094edcf30b","Type":"ContainerStarted","Data":"3c2ca092b4284b697ec434273641b646fc4316776e4f74977434ffd790f6bb3c"} Jul 28 13:44:24.256853 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.256829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" event={"ID":"24274456-35d5-411f-a0a3-bad2157ea5a6","Type":"ContainerStarted","Data":"533180a6b09072da3b968bb03e940e00893c2e80efd911b7b8a11f4b42ad578f"} Jul 28 13:44:24.276383 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.276327 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j6d4s" podStartSLOduration=32.482595358 podStartE2EDuration="35.276312447s" totalImagesPullingTime="2.793717089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:24.275358933 +0000 UTC m=+115.976902298" watchObservedRunningTime="2026-07-28 13:44:24.276312447 +0000 UTC m=+115.977855774" Jul 28 13:44:24.307006 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.306953 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-sg7c6" podStartSLOduration=32.430559421 podStartE2EDuration="35.306937912s" totalImagesPullingTime="2.876378491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:24.306527749 +0000 UTC m=+116.008071089" watchObservedRunningTime="2026-07-28 13:44:24.306937912 +0000 UTC m=+116.008481238" Jul 28 13:44:24.327694 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.327644 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bc86f4f56-k9bx6" podStartSLOduration=2.835431633 podStartE2EDuration="6.327627598s" totalImagesPullingTime="3.492195965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:24.327084128 +0000 UTC m=+116.028627456" watchObservedRunningTime="2026-07-28 13:44:24.327627598 +0000 UTC m=+116.029170925" Jul 28 13:44:24.587804 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.587706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:24.589998 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.589969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/660b57a2-689c-49b9-a48d-5bebc4138ff5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxwmm\" (UID: \"660b57a2-689c-49b9-a48d-5bebc4138ff5\") " pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:24.877883 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:24.877771 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lxwmm" Jul 28 13:44:25.008573 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:25.008544 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lxwmm"] Jul 28 13:44:25.011913 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:25.011878 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660b57a2_689c_49b9_a48d_5bebc4138ff5.slice/crio-2f7cd4b77fea50b29f7a366cd00129c9edff47bc11c12cc7ff8cc2bb57245fde WatchSource:0}: Error finding container 2f7cd4b77fea50b29f7a366cd00129c9edff47bc11c12cc7ff8cc2bb57245fde: Status 404 returned error can't find the container with id 2f7cd4b77fea50b29f7a366cd00129c9edff47bc11c12cc7ff8cc2bb57245fde Jul 28 13:44:25.261509 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:25.261458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxwmm" event={"ID":"660b57a2-689c-49b9-a48d-5bebc4138ff5","Type":"ContainerStarted","Data":"2f7cd4b77fea50b29f7a366cd00129c9edff47bc11c12cc7ff8cc2bb57245fde"} Jul 28 13:44:28.569421 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:28.569396 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:44:30.280529 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:30.280494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxwmm" event={"ID":"660b57a2-689c-49b9-a48d-5bebc4138ff5","Type":"ContainerStarted","Data":"1acc8b6b8e212880feab2aaab09b8d0555c2c4000556b72d8e0544f498c1c13f"} Jul 28 13:44:34.297898 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:34.297858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxwmm" event={"ID":"660b57a2-689c-49b9-a48d-5bebc4138ff5","Type":"ContainerStarted","Data":"460b0ec5026ed5a33a89b216901f85651dc47a024a90dfb53efcc5609c86ec09"} Jul 28 13:44:37.097844 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.097807 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx"] Jul 28 13:44:37.120025 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.119994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.126022 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.125988 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx"] Jul 28 13:44:37.127093 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.126873 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v9rk4\"" Jul 28 13:44:37.127500 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.127420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 13:44:37.128747 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.128722 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 13:44:37.129170 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.129127 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 13:44:37.130926 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.130869 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw"] Jul 28 13:44:37.136990 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.136971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.148100 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.148078 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 13:44:37.148277 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.148238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kn7ch\"" Jul 28 13:44:37.151610 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.151572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 13:44:37.151705 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.151572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 13:44:37.157279 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.157259 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw"] Jul 28 13:44:37.191374 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.191542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.191542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/514a877e-f2e4-4330-9206-eb8dd8fd20d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.191542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/735b6e9e-3349-4c06-8277-eb78764914b9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.191542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514a877e-f2e4-4330-9206-eb8dd8fd20d9-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.191542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcj8\" (UniqueName: \"kubernetes.io/projected/514a877e-f2e4-4330-9206-eb8dd8fd20d9-kube-api-access-8tcj8\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.191783 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735b6e9e-3349-4c06-8277-eb78764914b9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.191783 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/514a877e-f2e4-4330-9206-eb8dd8fd20d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.191783 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.191783 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.191677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnd2\" (UniqueName: \"kubernetes.io/projected/735b6e9e-3349-4c06-8277-eb78764914b9-kube-api-access-rqnd2\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.194457 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.194432 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w58qm"] Jul 28 13:44:37.199961 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.199940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.202954 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.202931 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 13:44:37.203067 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.203012 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 13:44:37.203129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.203065 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qbfrr\"" Jul 28 13:44:37.203287 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.203270 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 13:44:37.292445 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514a877e-f2e4-4330-9206-eb8dd8fd20d9-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.292445 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcj8\" (UniqueName: \"kubernetes.io/projected/514a877e-f2e4-4330-9206-eb8dd8fd20d9-kube-api-access-8tcj8\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.292716 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.292716 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-textfile\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.292716 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-tls\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.292716 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735b6e9e-3349-4c06-8277-eb78764914b9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.292716 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/514a877e-f2e4-4330-9206-eb8dd8fd20d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.292716 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.292716 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.293054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnd2\" (UniqueName: \"kubernetes.io/projected/735b6e9e-3349-4c06-8277-eb78764914b9-kube-api-access-rqnd2\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.293054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.293054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.293054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08f147d7-5099-4a2d-a918-d92893c355ff-metrics-client-ca\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.293054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4n79\" (UniqueName: \"kubernetes.io/projected/08f147d7-5099-4a2d-a918-d92893c355ff-kube-api-access-s4n79\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.293054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.292958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-root\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.293054 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-wtmp\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.293428 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/514a877e-f2e4-4330-9206-eb8dd8fd20d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.293428 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/735b6e9e-3349-4c06-8277-eb78764914b9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.293428 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-sys\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.293428 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:37.293135 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 28 13:44:37.293428 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:37.293194 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-tls podName:735b6e9e-3349-4c06-8277-eb78764914b9 nodeName:}" failed. No retries permitted until 2026-07-28 13:44:37.793175073 +0000 UTC m=+129.494718384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-xvzcw" (UID: "735b6e9e-3349-4c06-8277-eb78764914b9") : secret "kube-state-metrics-tls" not found Jul 28 13:44:37.293428 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/514a877e-f2e4-4330-9206-eb8dd8fd20d9-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.293428 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293339 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735b6e9e-3349-4c06-8277-eb78764914b9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.293772 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/735b6e9e-3349-4c06-8277-eb78764914b9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.293825 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.293780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.295688 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.295642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/514a877e-f2e4-4330-9206-eb8dd8fd20d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.295812 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.295753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.295812 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.295784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/514a877e-f2e4-4330-9206-eb8dd8fd20d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.305846 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.305812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcj8\" (UniqueName: \"kubernetes.io/projected/514a877e-f2e4-4330-9206-eb8dd8fd20d9-kube-api-access-8tcj8\") pod \"openshift-state-metrics-b8f49f5c6-rhprx\" (UID: \"514a877e-f2e4-4330-9206-eb8dd8fd20d9\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.314551 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.314512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnd2\" (UniqueName: \"kubernetes.io/projected/735b6e9e-3349-4c06-8277-eb78764914b9-kube-api-access-rqnd2\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.394168 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-wtmp\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394168 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-sys\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394168 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394422 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-textfile\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394422 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-tls\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394422 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394422 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-wtmp\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394422 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08f147d7-5099-4a2d-a918-d92893c355ff-metrics-client-ca\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394422 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4n79\" (UniqueName: \"kubernetes.io/projected/08f147d7-5099-4a2d-a918-d92893c355ff-kube-api-access-s4n79\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394422 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-root\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394746 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-root\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394746 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08f147d7-5099-4a2d-a918-d92893c355ff-sys\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.394953 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.394931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-textfile\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.395037 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.395024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.395188 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.395143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08f147d7-5099-4a2d-a918-d92893c355ff-metrics-client-ca\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.397214 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.397187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-tls\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.397214 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.397209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08f147d7-5099-4a2d-a918-d92893c355ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.405196 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.405165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4n79\" (UniqueName: \"kubernetes.io/projected/08f147d7-5099-4a2d-a918-d92893c355ff-kube-api-access-s4n79\") pod \"node-exporter-w58qm\" (UID: \"08f147d7-5099-4a2d-a918-d92893c355ff\") " pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.433042 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.433014 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" Jul 28 13:44:37.511752 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.511717 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w58qm" Jul 28 13:44:37.552956 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:37.552911 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f147d7_5099_4a2d_a918_d92893c355ff.slice/crio-3dd66a1e908d6a6b1c0ed66e63f99b88fb435dc642aaebba26fc93e604b8fd00 WatchSource:0}: Error finding container 3dd66a1e908d6a6b1c0ed66e63f99b88fb435dc642aaebba26fc93e604b8fd00: Status 404 returned error can't find the container with id 3dd66a1e908d6a6b1c0ed66e63f99b88fb435dc642aaebba26fc93e604b8fd00 Jul 28 13:44:37.641445 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.641422 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx"] Jul 28 13:44:37.644056 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:37.644028 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514a877e_f2e4_4330_9206_eb8dd8fd20d9.slice/crio-8d587ad7dd5829090090717119adedf0026405b876404d15ad7f394d92914167 WatchSource:0}: Error finding container 8d587ad7dd5829090090717119adedf0026405b876404d15ad7f394d92914167: Status 404 returned error can't find the container with id 8d587ad7dd5829090090717119adedf0026405b876404d15ad7f394d92914167 Jul 28 13:44:37.798356 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.798271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:37.800643 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:37.800618 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735b6e9e-3349-4c06-8277-eb78764914b9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xvzcw\" (UID: \"735b6e9e-3349-4c06-8277-eb78764914b9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:38.050969 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.050932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" Jul 28 13:44:38.195524 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.195495 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw"] Jul 28 13:44:38.197548 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:38.197500 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735b6e9e_3349_4c06_8277_eb78764914b9.slice/crio-eb984c3f28a6edec65d74e8799bc162f5d82c79bd27914c80dbcd5a2b67405e7 WatchSource:0}: Error finding container eb984c3f28a6edec65d74e8799bc162f5d82c79bd27914c80dbcd5a2b67405e7: Status 404 returned error can't find the container with id eb984c3f28a6edec65d74e8799bc162f5d82c79bd27914c80dbcd5a2b67405e7 Jul 28 13:44:38.315997 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.315860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxwmm" event={"ID":"660b57a2-689c-49b9-a48d-5bebc4138ff5","Type":"ContainerStarted","Data":"e513735048742fde797c2cb577ceee9154aa13f9c4f7ea3f3135a39bc9ec5250"} Jul 28 13:44:38.317928 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.317894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" event={"ID":"735b6e9e-3349-4c06-8277-eb78764914b9","Type":"ContainerStarted","Data":"eb984c3f28a6edec65d74e8799bc162f5d82c79bd27914c80dbcd5a2b67405e7"} Jul 28 13:44:38.320116 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.320086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" event={"ID":"514a877e-f2e4-4330-9206-eb8dd8fd20d9","Type":"ContainerStarted","Data":"316656f84b27459360fafcea6f82c2f1da0ae5980003ee9e307326dd3d33e2c1"} Jul 28 13:44:38.320233 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.320145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" event={"ID":"514a877e-f2e4-4330-9206-eb8dd8fd20d9","Type":"ContainerStarted","Data":"49523ae19e197f6e26136bbca948da776c85c6f3e4c6bf7cab0aa7fd51d840d9"} Jul 28 13:44:38.320233 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.320174 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" event={"ID":"514a877e-f2e4-4330-9206-eb8dd8fd20d9","Type":"ContainerStarted","Data":"8d587ad7dd5829090090717119adedf0026405b876404d15ad7f394d92914167"} Jul 28 13:44:38.321339 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.321311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w58qm" event={"ID":"08f147d7-5099-4a2d-a918-d92893c355ff","Type":"ContainerStarted","Data":"3dd66a1e908d6a6b1c0ed66e63f99b88fb435dc642aaebba26fc93e604b8fd00"} Jul 28 13:44:38.334176 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.334099 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lxwmm" podStartSLOduration=33.896924613 podStartE2EDuration="46.334079372s" totalImagesPullingTime="12.437154759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 13:43:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:38.333907561 +0000 UTC m=+130.035450889" watchObservedRunningTime="2026-07-28 13:44:38.334079372 +0000 UTC m=+130.035622701" Jul 28 13:44:38.505035 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.504998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:44:38.507690 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.507664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f6e6450-cd8c-4979-90e5-06011a1d268c-metrics-certs\") pod \"network-metrics-daemon-k4k9j\" (UID: \"6f6e6450-cd8c-4979-90e5-06011a1d268c\") " pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:44:38.590893 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.590873 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7bxcb\"" Jul 28 13:44:38.598459 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.598408 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k4k9j" Jul 28 13:44:38.738292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:38.738260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k4k9j"] Jul 28 13:44:38.741775 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:38.741747 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6e6450_cd8c_4979_90e5_06011a1d268c.slice/crio-9cf54189b4fbc4b044f2fd75245323022ccecb2622543ef9f2972fa71c161a5d WatchSource:0}: Error finding container 9cf54189b4fbc4b044f2fd75245323022ccecb2622543ef9f2972fa71c161a5d: Status 404 returned error can't find the container with id 9cf54189b4fbc4b044f2fd75245323022ccecb2622543ef9f2972fa71c161a5d Jul 28 13:44:39.327903 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:39.327702 2564 generic.go:356] "Generic (PLEG): container finished" podID="08f147d7-5099-4a2d-a918-d92893c355ff" containerID="cc1825ac18412938f011fead8685b48b4a6a66b05d4bf5e17ce43111faa8abe5" exitCode=0 Jul 28 13:44:39.327903 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:39.327830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w58qm" event={"ID":"08f147d7-5099-4a2d-a918-d92893c355ff","Type":"ContainerDied","Data":"cc1825ac18412938f011fead8685b48b4a6a66b05d4bf5e17ce43111faa8abe5"} Jul 28 13:44:39.329803 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:39.329729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k4k9j" event={"ID":"6f6e6450-cd8c-4979-90e5-06011a1d268c","Type":"ContainerStarted","Data":"9cf54189b4fbc4b044f2fd75245323022ccecb2622543ef9f2972fa71c161a5d"} Jul 28 13:44:40.242807 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.242201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-859d744f44-476ss" Jul 28 13:44:40.339252 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.337973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" event={"ID":"735b6e9e-3349-4c06-8277-eb78764914b9","Type":"ContainerStarted","Data":"c848c3ac9255991b938380c59d2e8fde08b84184071cb8cb6ce7685b2fb988a1"} Jul 28 13:44:40.339252 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.338053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" event={"ID":"735b6e9e-3349-4c06-8277-eb78764914b9","Type":"ContainerStarted","Data":"8b1539a92254ecaab8f6051696f74a66b97c953e22780eb1da2e1f39d6d922cb"} Jul 28 13:44:40.339252 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.338088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" event={"ID":"735b6e9e-3349-4c06-8277-eb78764914b9","Type":"ContainerStarted","Data":"a32c97a4e2a48d11411dd5b8177c6db12576390286d618156959c4bc0139783f"} Jul 28 13:44:40.341806 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.341753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" event={"ID":"514a877e-f2e4-4330-9206-eb8dd8fd20d9","Type":"ContainerStarted","Data":"59807afe490a75f2dca41b82186b851e96d92fb2eefdb76f20275b00ad2c55eb"} Jul 28 13:44:40.345220 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.345184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w58qm" event={"ID":"08f147d7-5099-4a2d-a918-d92893c355ff","Type":"ContainerStarted","Data":"284e913698847bca5c79e24ffecad612604906a13be6a90efca6c0d0a53a01be"} Jul 28 13:44:40.345334 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.345243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w58qm" event={"ID":"08f147d7-5099-4a2d-a918-d92893c355ff","Type":"ContainerStarted","Data":"938ed78b40958b69ffa1b68b5e369de4c0b5a4a15516699833419d2fee4d23f4"} Jul 28 13:44:40.347125 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.347102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k4k9j" event={"ID":"6f6e6450-cd8c-4979-90e5-06011a1d268c","Type":"ContainerStarted","Data":"ea613ed9bd68490c6721b53dc5e5b469e770f3bc9121f40e52486f41e96b0632"} Jul 28 13:44:40.347239 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.347143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k4k9j" event={"ID":"6f6e6450-cd8c-4979-90e5-06011a1d268c","Type":"ContainerStarted","Data":"aa55e2b5529b4111fbf00334049132a8e122dc19e3965dd692b881a4096019c0"} Jul 28 13:44:40.363522 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.363473 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xvzcw" podStartSLOduration=1.656863173 podStartE2EDuration="3.363458997s" totalImagesPullingTime="1.706595824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:40.361854118 +0000 UTC m=+132.063397444" watchObservedRunningTime="2026-07-28 13:44:40.363458997 +0000 UTC m=+132.065002316" Jul 28 13:44:40.389049 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.388980 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rhprx" podStartSLOduration=1.257654192 podStartE2EDuration="3.388962282s" totalImagesPullingTime="2.13130809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:40.388221915 +0000 UTC m=+132.089765260" watchObservedRunningTime="2026-07-28 13:44:40.388962282 +0000 UTC m=+132.090505618" Jul 28 13:44:40.408983 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.408925 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k4k9j" podStartSLOduration=130.079982988 podStartE2EDuration="2m11.40890899s" totalImagesPullingTime="1.328926002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:40.408070336 +0000 UTC m=+132.109613662" watchObservedRunningTime="2026-07-28 13:44:40.40890899 +0000 UTC m=+132.110452318" Jul 28 13:44:40.429028 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:40.428919 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w58qm" podStartSLOduration=2.321979923 podStartE2EDuration="3.428901623s" totalImagesPullingTime="1.037789778s" totalInitContainerRuntime="69.131922ms" isStatefulPod=true podCreationTimestamp="2026-07-28 13:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:40.428495343 +0000 UTC m=+132.130038729" watchObservedRunningTime="2026-07-28 13:44:40.428901623 +0000 UTC m=+132.130444952" Jul 28 13:44:41.711076 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.711041 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg"] Jul 28 13:44:41.714326 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.714306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:41.716736 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.716715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 13:44:41.716870 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.716853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6q9jm\"" Jul 28 13:44:41.723743 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.723720 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg"] Jul 28 13:44:41.838171 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.838133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/420193a6-0784-4014-aa0a-8a35eba222bc-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2g5vg\" (UID: \"420193a6-0784-4014-aa0a-8a35eba222bc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:41.838375 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.838184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whsnx\" (UniqueName: \"kubernetes.io/projected/420193a6-0784-4014-aa0a-8a35eba222bc-kube-api-access-whsnx\") pod \"monitoring-plugin-68b7fc9574-2g5vg\" (UID: \"420193a6-0784-4014-aa0a-8a35eba222bc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:41.939326 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.939282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whsnx\" (UniqueName: \"kubernetes.io/projected/420193a6-0784-4014-aa0a-8a35eba222bc-kube-api-access-whsnx\") pod \"monitoring-plugin-68b7fc9574-2g5vg\" (UID: \"420193a6-0784-4014-aa0a-8a35eba222bc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:41.939490 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.939379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/420193a6-0784-4014-aa0a-8a35eba222bc-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2g5vg\" (UID: \"420193a6-0784-4014-aa0a-8a35eba222bc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:41.941809 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.941785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/420193a6-0784-4014-aa0a-8a35eba222bc-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2g5vg\" (UID: \"420193a6-0784-4014-aa0a-8a35eba222bc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:41.949266 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:41.949237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whsnx\" (UniqueName: \"kubernetes.io/projected/420193a6-0784-4014-aa0a-8a35eba222bc-kube-api-access-whsnx\") pod \"monitoring-plugin-68b7fc9574-2g5vg\" (UID: \"420193a6-0784-4014-aa0a-8a35eba222bc\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:42.024562 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:42.024526 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:42.156409 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:42.156367 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg"] Jul 28 13:44:42.158628 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:42.158601 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420193a6_0784_4014_aa0a_8a35eba222bc.slice/crio-db232331979dacfceb32fcdb1bd9ba58b2171c58bfc349141604246380bfedb8 WatchSource:0}: Error finding container db232331979dacfceb32fcdb1bd9ba58b2171c58bfc349141604246380bfedb8: Status 404 returned error can't find the container with id db232331979dacfceb32fcdb1bd9ba58b2171c58bfc349141604246380bfedb8 Jul 28 13:44:42.356243 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:42.356153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" event={"ID":"420193a6-0784-4014-aa0a-8a35eba222bc","Type":"ContainerStarted","Data":"db232331979dacfceb32fcdb1bd9ba58b2171c58bfc349141604246380bfedb8"} Jul 28 13:44:43.582618 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.582558 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" podUID="ea000a28-6cc0-4fc0-9a95-6e005069082e" containerName="registry" containerID="cri-o://b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60" gracePeriod=30 Jul 28 13:44:43.830987 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.830958 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:44:43.954598 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954487 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-bound-sa-token\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.954598 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954545 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-installation-pull-secrets\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.954598 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954565 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-trusted-ca\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.954880 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954612 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea000a28-6cc0-4fc0-9a95-6e005069082e-ca-trust-extracted\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.954880 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954799 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.954880 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954862 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7d72\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-kube-api-access-w7d72\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.955033 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954901 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-image-registry-private-configuration\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.955033 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.954935 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-certificates\") pod \"ea000a28-6cc0-4fc0-9a95-6e005069082e\" (UID: \"ea000a28-6cc0-4fc0-9a95-6e005069082e\") " Jul 28 13:44:43.955129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.955067 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:44:43.955332 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.955204 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-trusted-ca\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:43.955458 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.955415 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:44:43.957268 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.957238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-kube-api-access-w7d72" (OuterVolumeSpecName: "kube-api-access-w7d72") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "kube-api-access-w7d72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:44:43.957365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.957262 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:44:43.957570 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.957512 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:44:43.957646 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.957567 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:44:43.957687 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.957641 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:44:43.963045 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:43.963022 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea000a28-6cc0-4fc0-9a95-6e005069082e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ea000a28-6cc0-4fc0-9a95-6e005069082e" (UID: "ea000a28-6cc0-4fc0-9a95-6e005069082e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:44:44.055675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.055639 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:44.055675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.055668 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7d72\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-kube-api-access-w7d72\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:44.055675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.055682 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-image-registry-private-configuration\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:44.055972 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.055695 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea000a28-6cc0-4fc0-9a95-6e005069082e-registry-certificates\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:44.055972 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.055706 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea000a28-6cc0-4fc0-9a95-6e005069082e-bound-sa-token\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:44.055972 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.055717 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea000a28-6cc0-4fc0-9a95-6e005069082e-installation-pull-secrets\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:44.055972 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.055730 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea000a28-6cc0-4fc0-9a95-6e005069082e-ca-trust-extracted\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:44:44.365965 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.365923 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" event={"ID":"420193a6-0784-4014-aa0a-8a35eba222bc","Type":"ContainerStarted","Data":"22fac1febda3f6cc0eb15dd4d56c692a849b485c4e1219dd642d9ea4093cabd2"} Jul 28 13:44:44.366189 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.366061 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:44.367231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.367175 2564 generic.go:356] "Generic (PLEG): container finished" podID="ea000a28-6cc0-4fc0-9a95-6e005069082e" containerID="b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60" exitCode=0 Jul 28 13:44:44.367231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.367213 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" Jul 28 13:44:44.367231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.367223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" event={"ID":"ea000a28-6cc0-4fc0-9a95-6e005069082e","Type":"ContainerDied","Data":"b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60"} Jul 28 13:44:44.367497 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.367266 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5464888c74-kr2h4" event={"ID":"ea000a28-6cc0-4fc0-9a95-6e005069082e","Type":"ContainerDied","Data":"f28eb14419c40d25134ebf89b950cc19128c46bd692d32e92bcaf24742ca1f30"} Jul 28 13:44:44.367497 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.367290 2564 scope.go:117] "RemoveContainer" containerID="b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60" Jul 28 13:44:44.371784 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.371761 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" Jul 28 13:44:44.376519 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.376505 2564 scope.go:117] "RemoveContainer" containerID="b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60" Jul 28 13:44:44.376820 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:44:44.376801 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60\": container with ID starting with b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60 not found: ID does not exist" containerID="b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60" Jul 28 13:44:44.376880 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.376831 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60"} err="failed to get container status \"b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60\": rpc error: code = NotFound desc = could not find container \"b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60\": container with ID starting with b11b965e880d59946b54feef900df0fb4ee8711bda599ac058d41407cc7a0b60 not found: ID does not exist" Jul 28 13:44:44.383354 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.383310 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2g5vg" podStartSLOduration=2.139736111 podStartE2EDuration="3.383298095s" totalImagesPullingTime="1.243561984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:44.382005587 +0000 UTC m=+136.083548935" watchObservedRunningTime="2026-07-28 13:44:44.383298095 +0000 UTC m=+136.084841422" Jul 28 13:44:44.414900 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.414866 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5464888c74-kr2h4"] Jul 28 13:44:44.424996 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.424968 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5464888c74-kr2h4"] Jul 28 13:44:44.694095 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.694014 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-krsgb"] Jul 28 13:44:44.694431 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.694312 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea000a28-6cc0-4fc0-9a95-6e005069082e" containerName="registry" Jul 28 13:44:44.694431 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.694324 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea000a28-6cc0-4fc0-9a95-6e005069082e" containerName="registry" Jul 28 13:44:44.694431 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.694375 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea000a28-6cc0-4fc0-9a95-6e005069082e" containerName="registry" Jul 28 13:44:44.698668 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.698645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-krsgb" Jul 28 13:44:44.701440 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.701408 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 13:44:44.701591 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.701408 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 13:44:44.701591 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.701495 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7q6l4\"" Jul 28 13:44:44.709706 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.709680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-krsgb"] Jul 28 13:44:44.760526 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.760494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85x4\" (UniqueName: \"kubernetes.io/projected/72bf710f-32e1-40ac-94dc-361188ca94aa-kube-api-access-b85x4\") pod \"downloads-7c97dc4555-krsgb\" (UID: \"72bf710f-32e1-40ac-94dc-361188ca94aa\") " pod="openshift-console/downloads-7c97dc4555-krsgb" Jul 28 13:44:44.782214 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.782180 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea000a28-6cc0-4fc0-9a95-6e005069082e" path="/var/lib/kubelet/pods/ea000a28-6cc0-4fc0-9a95-6e005069082e/volumes" Jul 28 13:44:44.860912 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.860879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85x4\" (UniqueName: \"kubernetes.io/projected/72bf710f-32e1-40ac-94dc-361188ca94aa-kube-api-access-b85x4\") pod \"downloads-7c97dc4555-krsgb\" (UID: \"72bf710f-32e1-40ac-94dc-361188ca94aa\") " pod="openshift-console/downloads-7c97dc4555-krsgb" Jul 28 13:44:44.874736 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:44.874702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85x4\" (UniqueName: \"kubernetes.io/projected/72bf710f-32e1-40ac-94dc-361188ca94aa-kube-api-access-b85x4\") pod \"downloads-7c97dc4555-krsgb\" (UID: \"72bf710f-32e1-40ac-94dc-361188ca94aa\") " pod="openshift-console/downloads-7c97dc4555-krsgb" Jul 28 13:44:45.008173 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:45.008141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-krsgb" Jul 28 13:44:45.152057 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:45.152022 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-krsgb"] Jul 28 13:44:45.155934 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:44:45.155903 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bf710f_32e1_40ac_94dc_361188ca94aa.slice/crio-def840c8ec0323c7f4570177689e8d0dc0bea3b5edd312dacc997eaf8c1b741b WatchSource:0}: Error finding container def840c8ec0323c7f4570177689e8d0dc0bea3b5edd312dacc997eaf8c1b741b: Status 404 returned error can't find the container with id def840c8ec0323c7f4570177689e8d0dc0bea3b5edd312dacc997eaf8c1b741b Jul 28 13:44:45.371713 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:45.371612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-krsgb" event={"ID":"72bf710f-32e1-40ac-94dc-361188ca94aa","Type":"ContainerStarted","Data":"def840c8ec0323c7f4570177689e8d0dc0bea3b5edd312dacc997eaf8c1b741b"} Jul 28 13:44:55.806244 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.806213 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fb4566bd-bt2pt"] Jul 28 13:44:55.811728 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.811701 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:55.814610 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.814570 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 13:44:55.814610 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.814599 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 13:44:55.815940 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.815719 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 13:44:55.815940 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.815738 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 13:44:55.815940 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.815798 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c2kc8\"" Jul 28 13:44:55.815940 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.815721 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 13:44:55.820151 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.820120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb4566bd-bt2pt"] Jul 28 13:44:55.958159 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.958116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-oauth-config\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:55.958339 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.958177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5w4c\" (UniqueName: \"kubernetes.io/projected/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-kube-api-access-z5w4c\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:55.958339 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.958213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-oauth-serving-cert\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:55.958339 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.958246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-config\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:55.958339 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.958268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-serving-cert\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:55.958339 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:55.958295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-service-ca\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.058966 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.058870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5w4c\" (UniqueName: \"kubernetes.io/projected/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-kube-api-access-z5w4c\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.058966 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.058915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-oauth-serving-cert\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.058966 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.058947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-config\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.059247 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.058986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-serving-cert\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.059247 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.059021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-service-ca\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.059247 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.059050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-oauth-config\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.059915 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.059809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-oauth-serving-cert\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.059915 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.059828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-service-ca\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.059915 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.059854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-config\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.061845 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.061824 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-serving-cert\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.061935 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.061904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-oauth-config\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.069768 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.069714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5w4c\" (UniqueName: \"kubernetes.io/projected/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-kube-api-access-z5w4c\") pod \"console-5fb4566bd-bt2pt\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:44:56.124096 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:44:56.124058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:45:01.816916 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:01.816889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb4566bd-bt2pt"] Jul 28 13:45:01.967507 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:45:01.967432 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa289dc0_2b5d_4df7_b475_0bc5fb19bb6c.slice/crio-a1c93b7260d8677d2f6ba7d6b7d6418d0f2116d3c1ae782521fb2deb7d551c2c WatchSource:0}: Error finding container a1c93b7260d8677d2f6ba7d6b7d6418d0f2116d3c1ae782521fb2deb7d551c2c: Status 404 returned error can't find the container with id a1c93b7260d8677d2f6ba7d6b7d6418d0f2116d3c1ae782521fb2deb7d551c2c Jul 28 13:45:02.446712 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:02.446378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-krsgb" event={"ID":"72bf710f-32e1-40ac-94dc-361188ca94aa","Type":"ContainerStarted","Data":"fa339d1242708f2839c1b064df29c685a4bc07062cc6529cdee34ce0a8736710"} Jul 28 13:45:02.446712 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:02.446450 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-krsgb" Jul 28 13:45:02.447896 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:02.447865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4566bd-bt2pt" event={"ID":"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c","Type":"ContainerStarted","Data":"a1c93b7260d8677d2f6ba7d6b7d6418d0f2116d3c1ae782521fb2deb7d551c2c"} Jul 28 13:45:02.448084 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:02.448061 2564 patch_prober.go:28] interesting pod/downloads-7c97dc4555-krsgb container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused" start-of-body= Jul 28 13:45:02.448143 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:02.448119 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-krsgb" podUID="72bf710f-32e1-40ac-94dc-361188ca94aa" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused" Jul 28 13:45:02.471482 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:02.471431 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-krsgb" podStartSLOduration=1.583498525 podStartE2EDuration="18.471413435s" totalImagesPullingTime="16.88791491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:45:02.469616976 +0000 UTC m=+154.171160321" watchObservedRunningTime="2026-07-28 13:45:02.471413435 +0000 UTC m=+154.172956761" Jul 28 13:45:03.463830 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:03.463767 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-krsgb" Jul 28 13:45:07.478110 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:07.478011 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4566bd-bt2pt" event={"ID":"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c","Type":"ContainerStarted","Data":"3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0"} Jul 28 13:45:07.504903 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:07.504849 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fb4566bd-bt2pt" podStartSLOduration=7.962655189 podStartE2EDuration="12.504829666s" totalImagesPullingTime="4.542174477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:45:07.503573511 +0000 UTC m=+159.205116838" watchObservedRunningTime="2026-07-28 13:45:07.504829666 +0000 UTC m=+159.206372996" Jul 28 13:45:12.499015 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:12.498978 2564 generic.go:356] "Generic (PLEG): container finished" podID="bd6104f8-624a-41ec-bda3-7ccba6de3105" containerID="00a318479f11b2d993f7549d3d42e2199db97d9d9168a3dfa6b36d68b334b83b" exitCode=0 Jul 28 13:45:12.499513 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:12.499025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" event={"ID":"bd6104f8-624a-41ec-bda3-7ccba6de3105","Type":"ContainerDied","Data":"00a318479f11b2d993f7549d3d42e2199db97d9d9168a3dfa6b36d68b334b83b"} Jul 28 13:45:12.499513 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:12.499441 2564 scope.go:117] "RemoveContainer" containerID="00a318479f11b2d993f7549d3d42e2199db97d9d9168a3dfa6b36d68b334b83b" Jul 28 13:45:13.505118 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:13.505083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-zmts5" event={"ID":"bd6104f8-624a-41ec-bda3-7ccba6de3105","Type":"ContainerStarted","Data":"039cfcc0836ffc83a0f4a1ee94c97d63d4fe2d14022519665f29a73a82802986"} Jul 28 13:45:14.964695 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:14.964657 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fb4566bd-bt2pt"] Jul 28 13:45:16.125187 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:16.125147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:45:39.983476 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:39.983414 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fb4566bd-bt2pt" podUID="fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" containerName="console" containerID="cri-o://3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0" gracePeriod=15 Jul 28 13:45:40.257655 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.257529 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:45:40.319679 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.319630 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-oauth-config\") pod \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " Jul 28 13:45:40.319851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.319710 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-config\") pod \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " Jul 28 13:45:40.319851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.319751 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5w4c\" (UniqueName: \"kubernetes.io/projected/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-kube-api-access-z5w4c\") pod \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " Jul 28 13:45:40.319851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.319773 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-serving-cert\") pod \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " Jul 28 13:45:40.319851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.319796 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-service-ca\") pod \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " Jul 28 13:45:40.320051 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.319852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-oauth-serving-cert\") pod \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\" (UID: \"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c\") " Jul 28 13:45:40.320224 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.320190 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-config" (OuterVolumeSpecName: "console-config") pod "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" (UID: "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:45:40.320365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.320215 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-service-ca" (OuterVolumeSpecName: "service-ca") pod "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" (UID: "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:45:40.320365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.320239 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" (UID: "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:45:40.322012 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.321988 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" (UID: "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:45:40.322381 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.322356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" (UID: "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:45:40.322381 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.322366 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-kube-api-access-z5w4c" (OuterVolumeSpecName: "kube-api-access-z5w4c") pod "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" (UID: "fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c"). InnerVolumeSpecName "kube-api-access-z5w4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:45:40.420949 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.420905 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-service-ca\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:45:40.420949 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.420943 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-oauth-serving-cert\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:45:40.420949 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.420957 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-oauth-config\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:45:40.421187 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.420969 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-config\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:45:40.421187 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.420982 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5w4c\" (UniqueName: \"kubernetes.io/projected/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-kube-api-access-z5w4c\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:45:40.421187 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.420993 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c-console-serving-cert\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:45:40.604229 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.604139 2564 generic.go:356] "Generic (PLEG): container finished" podID="fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" containerID="3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0" exitCode=0 Jul 28 13:45:40.604229 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.604200 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4566bd-bt2pt" Jul 28 13:45:40.604478 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.604204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4566bd-bt2pt" event={"ID":"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c","Type":"ContainerDied","Data":"3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0"} Jul 28 13:45:40.604478 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.604355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4566bd-bt2pt" event={"ID":"fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c","Type":"ContainerDied","Data":"a1c93b7260d8677d2f6ba7d6b7d6418d0f2116d3c1ae782521fb2deb7d551c2c"} Jul 28 13:45:40.604478 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.604376 2564 scope.go:117] "RemoveContainer" containerID="3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0" Jul 28 13:45:40.613187 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.613166 2564 scope.go:117] "RemoveContainer" containerID="3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0" Jul 28 13:45:40.613468 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:45:40.613445 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0\": container with ID starting with 3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0 not found: ID does not exist" containerID="3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0" Jul 28 13:45:40.613537 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.613478 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0"} err="failed to get container status \"3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0\": rpc error: code = NotFound desc = could not find container \"3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0\": container with ID starting with 3de4d1733dc8fdb3fb3d3261cc2738765b6b267d0e6e2ce8f1cff5254230a2f0 not found: ID does not exist" Jul 28 13:45:40.627737 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.627705 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fb4566bd-bt2pt"] Jul 28 13:45:40.634911 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.634886 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fb4566bd-bt2pt"] Jul 28 13:45:40.782194 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:45:40.782159 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" path="/var/lib/kubelet/pods/fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c/volumes" Jul 28 13:46:01.644476 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.644443 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt"] Jul 28 13:46:01.644933 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.644730 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" containerName="console" Jul 28 13:46:01.644933 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.644742 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" containerName="console" Jul 28 13:46:01.644933 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.644805 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa289dc0-2b5d-4df7-b475-0bc5fb19bb6c" containerName="console" Jul 28 13:46:01.647806 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.647787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.652191 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.652169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 13:46:01.652310 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.652174 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 13:46:01.652516 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.652497 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 13:46:01.653312 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.653292 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 13:46:01.653413 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.653364 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cjl7q\"" Jul 28 13:46:01.655260 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.655242 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 13:46:01.668237 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.668219 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 13:46:01.679664 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.679643 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt"] Jul 28 13:46:01.779641 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-federate-client-tls\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.779801 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.779801 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-serving-certs-ca-bundle\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.779801 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-secret-telemeter-client\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.779801 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-telemeter-client-tls\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.779801 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgnk\" (UniqueName: \"kubernetes.io/projected/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-kube-api-access-lzgnk\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.779801 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.779801 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.779768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-metrics-client-ca\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880575 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-serving-certs-ca-bundle\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880738 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-secret-telemeter-client\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880738 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-telemeter-client-tls\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880738 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgnk\" (UniqueName: \"kubernetes.io/projected/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-kube-api-access-lzgnk\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880866 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880866 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-metrics-client-ca\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880969 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-federate-client-tls\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.880969 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.880953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.881597 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.881558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-serving-certs-ca-bundle\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.881811 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.881785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.882001 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.881976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-metrics-client-ca\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.883316 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.883295 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-secret-telemeter-client\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.883408 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.883356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.883408 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.883382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-federate-client-tls\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.883543 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.883522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-telemeter-client-tls\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.892114 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.892096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgnk\" (UniqueName: \"kubernetes.io/projected/6ef0b607-e986-42d7-8e8c-6c5fd0557b93-kube-api-access-lzgnk\") pod \"telemeter-client-56cb6f8c64-8nrmt\" (UID: \"6ef0b607-e986-42d7-8e8c-6c5fd0557b93\") " pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:01.956510 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:01.956440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" Jul 28 13:46:02.096945 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:02.095771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt"] Jul 28 13:46:02.686720 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:02.686687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" event={"ID":"6ef0b607-e986-42d7-8e8c-6c5fd0557b93","Type":"ContainerStarted","Data":"358943c31af87db8e4e147817ba077fa5ac3b34e3c49a04da2672dc3417dd5b0"} Jul 28 13:46:04.694913 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:04.694853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" event={"ID":"6ef0b607-e986-42d7-8e8c-6c5fd0557b93","Type":"ContainerStarted","Data":"d42f40a27154d2d84c0a060291e68aacba0ed0b3c8a02b6f92871df3f56b3628"} Jul 28 13:46:06.706204 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:06.706178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" event={"ID":"6ef0b607-e986-42d7-8e8c-6c5fd0557b93","Type":"ContainerStarted","Data":"32cc6cc056e835bce63cba0b68182f6b14cdd38e3933cfad8b685623b448f81c"} Jul 28 13:46:06.706488 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:06.706234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" event={"ID":"6ef0b607-e986-42d7-8e8c-6c5fd0557b93","Type":"ContainerStarted","Data":"179708ab403e4bdb8668718bf933e9b2316cffc7e60255878a8e3cb6b6a46adb"} Jul 28 13:46:06.737673 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:46:06.737623 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56cb6f8c64-8nrmt" podStartSLOduration=1.321611264 podStartE2EDuration="5.737609011s" totalImagesPullingTime="4.415997747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:46:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:46:06.735652562 +0000 UTC m=+218.437195892" watchObservedRunningTime="2026-07-28 13:46:06.737609011 +0000 UTC m=+218.439152334" Jul 28 13:47:28.727970 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:47:28.727945 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 13:50:44.844926 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.844885 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762"] Jul 28 13:50:44.848712 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.848684 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:44.851785 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.851760 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-f82r5\"" Jul 28 13:50:44.851785 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.851780 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 13:50:44.851986 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.851781 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 28 13:50:44.851986 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.851800 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 28 13:50:44.851986 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.851807 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 28 13:50:44.852151 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.852135 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 13:50:44.852361 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.852343 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 28 13:50:44.852960 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.852947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 28 13:50:44.879672 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.879636 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762"] Jul 28 13:50:44.992862 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.992822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc92c226-960c-4314-b6d7-1138a64195a9-feast-data\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:44.992862 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.992864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-online-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:44.993118 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.992906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-registry-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:44.993118 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.992995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-offline-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:44.993118 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.993024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-ui-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:44.993118 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:44.993047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdp6q\" (UniqueName: \"kubernetes.io/projected/dc92c226-960c-4314-b6d7-1138a64195a9-kube-api-access-kdp6q\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.093773 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.093732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-offline-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.093773 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.093773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-ui-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.094032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.093795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdp6q\" (UniqueName: \"kubernetes.io/projected/dc92c226-960c-4314-b6d7-1138a64195a9-kube-api-access-kdp6q\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.094032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.093819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc92c226-960c-4314-b6d7-1138a64195a9-feast-data\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.094032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.093843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-online-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.094032 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.093887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-registry-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.094259 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.094233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc92c226-960c-4314-b6d7-1138a64195a9-feast-data\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.096460 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.096397 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-offline-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.096460 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.096427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-ui-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.096623 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.096603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-online-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.096665 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.096629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-registry-tls\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.105777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.105757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdp6q\" (UniqueName: \"kubernetes.io/projected/dc92c226-960c-4314-b6d7-1138a64195a9-kube-api-access-kdp6q\") pod \"feast-simple-feast-setup-c5f65bd88-nk762\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.158111 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.158060 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:50:45.300526 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.300488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762"] Jul 28 13:50:45.305376 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:50:45.305350 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc92c226_960c_4314_b6d7_1138a64195a9.slice/crio-8b3bb6e685627ce2593c060bad69e20ddb44e1ded65778319fe18f5dfdb02f8b WatchSource:0}: Error finding container 8b3bb6e685627ce2593c060bad69e20ddb44e1ded65778319fe18f5dfdb02f8b: Status 404 returned error can't find the container with id 8b3bb6e685627ce2593c060bad69e20ddb44e1ded65778319fe18f5dfdb02f8b Jul 28 13:50:45.307228 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.307207 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 13:50:45.707097 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:50:45.707062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerStarted","Data":"8b3bb6e685627ce2593c060bad69e20ddb44e1ded65778319fe18f5dfdb02f8b"} Jul 28 13:51:10.819472 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:10.819433 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerStarted","Data":"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22"} Jul 28 13:51:15.838976 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:15.838941 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc92c226-960c-4314-b6d7-1138a64195a9" containerID="07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22" exitCode=0 Jul 28 13:51:15.839566 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:15.839015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerDied","Data":"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22"} Jul 28 13:51:16.845094 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:16.845060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerStarted","Data":"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366"} Jul 28 13:51:21.867033 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:21.867000 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc92c226-960c-4314-b6d7-1138a64195a9" containerID="aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366" exitCode=0 Jul 28 13:51:21.867385 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:21.867076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerDied","Data":"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366"} Jul 28 13:51:22.877326 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:22.877281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerStarted","Data":"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720"} Jul 28 13:51:22.877828 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:22.877355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerStarted","Data":"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92"} Jul 28 13:51:22.877828 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:22.877378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerStarted","Data":"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370"} Jul 28 13:51:22.877828 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:22.877396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerStarted","Data":"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1"} Jul 28 13:51:22.929610 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:22.929302 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podStartSLOduration=14.030887936 podStartE2EDuration="38.929277843s" totalImagesPullingTime="24.898389907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:51:22.915768943 +0000 UTC m=+534.617312293" watchObservedRunningTime="2026-07-28 13:51:22.929277843 +0000 UTC m=+534.630821173" Jul 28 13:51:24.158427 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.158386 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:24.158936 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.158452 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:24.158936 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.158472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:24.158936 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.158493 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:24.161007 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.160971 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.22:6571: connect: connection refused" Jul 28 13:51:24.161271 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.161022 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.22:8816: connect: connection refused" Jul 28 13:51:24.161358 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.161035 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Jul 28 13:51:24.161358 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:24.161072 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Jul 28 13:51:25.158464 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:25.158420 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:25.158963 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:25.158491 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:25.158963 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:25.158510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:25.158963 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:25.158526 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:27.159346 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:27.159301 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.22:8816: connect: connection refused" Jul 28 13:51:27.160052 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:27.159301 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.22:6571: connect: connection refused" Jul 28 13:51:27.160190 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:27.159302 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Jul 28 13:51:27.160190 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:27.159356 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Jul 28 13:51:30.159360 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.159322 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:30.159898 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.159874 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:30.160010 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.159998 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:30.160171 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.160156 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:30.164485 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.164471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:30.919572 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.919545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:30.919778 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.919649 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:51:30.922289 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:51:30.922266 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:52:32.292249 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.292213 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762"] Jul 28 13:52:32.292849 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.292525 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="offline" containerID="cri-o://1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92" gracePeriod=30 Jul 28 13:52:32.292849 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.292559 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="registry" containerID="cri-o://b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1" gracePeriod=30 Jul 28 13:52:32.292849 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.292643 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" containerID="cri-o://f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370" gracePeriod=30 Jul 28 13:52:32.292849 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.292558 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" containerID="cri-o://4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720" gracePeriod=30 Jul 28 13:52:32.605308 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.605225 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2"] Jul 28 13:52:32.609052 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.609031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.618731 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.618703 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-tj6gl\"" Jul 28 13:52:32.654108 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.654072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2"] Jul 28 13:52:32.741012 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.740979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-registry-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.741231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.741033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-ui-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.741231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.741068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wpp\" (UniqueName: \"kubernetes.io/projected/6e2e54e2-15ef-4231-82ad-91e285f248d3-kube-api-access-h6wpp\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.741231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.741105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-online-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.741231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.741122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2e54e2-15ef-4231-82ad-91e285f248d3-feast-data\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.741231 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.741166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-offline-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.842331 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.842295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wpp\" (UniqueName: \"kubernetes.io/projected/6e2e54e2-15ef-4231-82ad-91e285f248d3-kube-api-access-h6wpp\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.842484 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.842462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-online-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.842548 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.842496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2e54e2-15ef-4231-82ad-91e285f248d3-feast-data\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.842548 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.842530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-offline-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.842665 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.842564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-registry-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.842665 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.842653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-ui-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.843024 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.843003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2e54e2-15ef-4231-82ad-91e285f248d3-feast-data\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.845099 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.845040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-online-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.845099 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.845055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-ui-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.845347 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.845326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-offline-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.845404 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.845356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-registry-tls\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.852635 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.852609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wpp\" (UniqueName: \"kubernetes.io/projected/6e2e54e2-15ef-4231-82ad-91e285f248d3-kube-api-access-h6wpp\") pod \"feast-simple-feast-setup-c5f65bd88-xzlg2\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:32.919498 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:32.919464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:33.077975 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:33.077947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2"] Jul 28 13:52:33.079802 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:52:33.079773 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e2e54e2_15ef_4231_82ad_91e285f248d3.slice/crio-3051f9c02c691f91d66ce99c0eebf256b507e2e526101cb90c3df80299936286 WatchSource:0}: Error finding container 3051f9c02c691f91d66ce99c0eebf256b507e2e526101cb90c3df80299936286: Status 404 returned error can't find the container with id 3051f9c02c691f91d66ce99c0eebf256b507e2e526101cb90c3df80299936286 Jul 28 13:52:33.169476 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:33.169387 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc92c226-960c-4314-b6d7-1138a64195a9" containerID="4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720" exitCode=0 Jul 28 13:52:33.169476 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:33.169422 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc92c226-960c-4314-b6d7-1138a64195a9" containerID="f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370" exitCode=0 Jul 28 13:52:33.169476 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:33.169450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerDied","Data":"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720"} Jul 28 13:52:33.169762 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:33.169494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerDied","Data":"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370"} Jul 28 13:52:33.171150 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:33.171117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerStarted","Data":"e3900de4ab03005a10b148024e94fdf7f63ddd4bc0146a0d7b310c15e5ee4bde"} Jul 28 13:52:33.171279 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:33.171169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerStarted","Data":"3051f9c02c691f91d66ce99c0eebf256b507e2e526101cb90c3df80299936286"} Jul 28 13:52:38.196912 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:38.196875 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerID="e3900de4ab03005a10b148024e94fdf7f63ddd4bc0146a0d7b310c15e5ee4bde" exitCode=0 Jul 28 13:52:38.197267 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:38.196950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerDied","Data":"e3900de4ab03005a10b148024e94fdf7f63ddd4bc0146a0d7b310c15e5ee4bde"} Jul 28 13:52:39.203376 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:39.203341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerStarted","Data":"f682e2300db0e9453c4c77a20268031dedac8223d207e99abcdf7eeeff841955"} Jul 28 13:52:40.919542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:40.919495 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Jul 28 13:52:40.919542 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:40.919522 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Jul 28 13:52:44.225240 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:44.225206 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerID="f682e2300db0e9453c4c77a20268031dedac8223d207e99abcdf7eeeff841955" exitCode=0 Jul 28 13:52:44.225641 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:44.225252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerDied","Data":"f682e2300db0e9453c4c77a20268031dedac8223d207e99abcdf7eeeff841955"} Jul 28 13:52:45.235136 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:45.235095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerStarted","Data":"386a1b0b528f0d4ae19183e0766926150b836c4908438016dbc184ca1948d1e1"} Jul 28 13:52:45.235593 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:45.235159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerStarted","Data":"cdd0b03212d32ab78e8f7c8aa908673be387309011fa27ca7abaaf2c2442e2d9"} Jul 28 13:52:45.235593 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:45.235179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerStarted","Data":"ba01635eef6b8b2c1d8f225ec3badd3f2f2ee6ea67e72e25abeda95128698467"} Jul 28 13:52:45.235593 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:45.235196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerStarted","Data":"61d82abd6971f057646b7dff15d586ec5fade02f3d6a40498d222501b626ca7a"} Jul 28 13:52:45.266711 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:45.266638 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podStartSLOduration=13.266614263 podStartE2EDuration="13.266614263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:52:45.263537484 +0000 UTC m=+616.965080839" watchObservedRunningTime="2026-07-28 13:52:45.266614263 +0000 UTC m=+616.968157591" Jul 28 13:52:47.920080 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.920037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:47.920594 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.920099 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:47.920594 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.920119 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:47.920594 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.920136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:47.922192 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.922153 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jul 28 13:52:47.922300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.922190 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jul 28 13:52:47.922300 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.922238 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.23:8816: connect: connection refused" Jul 28 13:52:47.922455 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:47.922178 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.23:6571: connect: connection refused" Jul 28 13:52:50.919614 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.919529 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Jul 28 13:52:50.919614 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.919526 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Jul 28 13:52:50.920663 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.920639 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.920775 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.920730 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.920970 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.920951 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.921297 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.921270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.921416 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.921314 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.921416 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.921336 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.921626 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.921606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.921703 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.921689 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.921746 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.921730 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:50.925656 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:50.925640 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:51.267385 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:51.267352 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:52:51.270628 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:52:51.270609 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:53:00.919697 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:00.919653 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" probeResult="failure" output="Get \"https://10.133.0.22:6567/health\": dial tcp 10.133.0.22:6567: connect: connection refused" Jul 28 13:53:00.920137 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:00.919652 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.22:8443: connect: connection refused" Jul 28 13:53:00.920137 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:00.919788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:53:00.920137 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:00.919850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:53:02.949486 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:02.949457 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:53:03.101558 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.101442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-registry-tls\") pod \"dc92c226-960c-4314-b6d7-1138a64195a9\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " Jul 28 13:53:03.101558 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.101500 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-ui-tls\") pod \"dc92c226-960c-4314-b6d7-1138a64195a9\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " Jul 28 13:53:03.101558 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.101552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc92c226-960c-4314-b6d7-1138a64195a9-feast-data\") pod \"dc92c226-960c-4314-b6d7-1138a64195a9\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " Jul 28 13:53:03.101871 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.101573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdp6q\" (UniqueName: \"kubernetes.io/projected/dc92c226-960c-4314-b6d7-1138a64195a9-kube-api-access-kdp6q\") pod \"dc92c226-960c-4314-b6d7-1138a64195a9\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " Jul 28 13:53:03.101871 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.101619 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-offline-tls\") pod \"dc92c226-960c-4314-b6d7-1138a64195a9\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " Jul 28 13:53:03.101871 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.101683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-online-tls\") pod \"dc92c226-960c-4314-b6d7-1138a64195a9\" (UID: \"dc92c226-960c-4314-b6d7-1138a64195a9\") " Jul 28 13:53:03.102148 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.102114 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc92c226-960c-4314-b6d7-1138a64195a9-feast-data" (OuterVolumeSpecName: "feast-data") pod "dc92c226-960c-4314-b6d7-1138a64195a9" (UID: "dc92c226-960c-4314-b6d7-1138a64195a9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:53:03.103952 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.103905 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-online-tls" (OuterVolumeSpecName: "online-tls") pod "dc92c226-960c-4314-b6d7-1138a64195a9" (UID: "dc92c226-960c-4314-b6d7-1138a64195a9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:53:03.104063 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.104015 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "dc92c226-960c-4314-b6d7-1138a64195a9" (UID: "dc92c226-960c-4314-b6d7-1138a64195a9"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:53:03.104063 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.104023 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc92c226-960c-4314-b6d7-1138a64195a9-kube-api-access-kdp6q" (OuterVolumeSpecName: "kube-api-access-kdp6q") pod "dc92c226-960c-4314-b6d7-1138a64195a9" (UID: "dc92c226-960c-4314-b6d7-1138a64195a9"). InnerVolumeSpecName "kube-api-access-kdp6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:53:03.104149 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.104083 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "dc92c226-960c-4314-b6d7-1138a64195a9" (UID: "dc92c226-960c-4314-b6d7-1138a64195a9"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:53:03.104251 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.104229 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dc92c226-960c-4314-b6d7-1138a64195a9" (UID: "dc92c226-960c-4314-b6d7-1138a64195a9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:53:03.202271 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.202233 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/dc92c226-960c-4314-b6d7-1138a64195a9-feast-data\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:53:03.202271 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.202266 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdp6q\" (UniqueName: \"kubernetes.io/projected/dc92c226-960c-4314-b6d7-1138a64195a9-kube-api-access-kdp6q\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:53:03.202271 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.202277 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-offline-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:53:03.202500 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.202286 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-online-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:53:03.202500 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.202293 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-registry-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:53:03.202500 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.202301 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/dc92c226-960c-4314-b6d7-1138a64195a9-ui-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:53:03.319176 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.319141 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc92c226-960c-4314-b6d7-1138a64195a9" containerID="1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92" exitCode=137 Jul 28 13:53:03.319176 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.319169 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc92c226-960c-4314-b6d7-1138a64195a9" containerID="b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1" exitCode=137 Jul 28 13:53:03.319439 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.319213 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" Jul 28 13:53:03.319439 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.319231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerDied","Data":"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92"} Jul 28 13:53:03.319439 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.319268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerDied","Data":"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1"} Jul 28 13:53:03.319439 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.319281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762" event={"ID":"dc92c226-960c-4314-b6d7-1138a64195a9","Type":"ContainerDied","Data":"8b3bb6e685627ce2593c060bad69e20ddb44e1ded65778319fe18f5dfdb02f8b"} Jul 28 13:53:03.319439 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.319298 2564 scope.go:117] "RemoveContainer" containerID="4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720" Jul 28 13:53:03.328855 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.328836 2564 scope.go:117] "RemoveContainer" containerID="1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92" Jul 28 13:53:03.336721 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.336700 2564 scope.go:117] "RemoveContainer" containerID="f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370" Jul 28 13:53:03.342351 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.342327 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762"] Jul 28 13:53:03.345868 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.345848 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-nk762"] Jul 28 13:53:03.345948 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.345928 2564 scope.go:117] "RemoveContainer" containerID="b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1" Jul 28 13:53:03.353988 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.353967 2564 scope.go:117] "RemoveContainer" containerID="aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366" Jul 28 13:53:03.361502 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.361482 2564 scope.go:117] "RemoveContainer" containerID="07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22" Jul 28 13:53:03.369006 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.368983 2564 scope.go:117] "RemoveContainer" containerID="4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720" Jul 28 13:53:03.369287 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:53:03.369269 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720\": container with ID starting with 4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720 not found: ID does not exist" containerID="4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720" Jul 28 13:53:03.369356 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.369295 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720"} err="failed to get container status \"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720\": rpc error: code = NotFound desc = could not find container \"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720\": container with ID starting with 4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720 not found: ID does not exist" Jul 28 13:53:03.369356 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.369315 2564 scope.go:117] "RemoveContainer" containerID="1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92" Jul 28 13:53:03.369561 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:53:03.369542 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92\": container with ID starting with 1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92 not found: ID does not exist" containerID="1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92" Jul 28 13:53:03.369654 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.369568 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92"} err="failed to get container status \"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92\": rpc error: code = NotFound desc = could not find container \"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92\": container with ID starting with 1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92 not found: ID does not exist" Jul 28 13:53:03.369654 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.369599 2564 scope.go:117] "RemoveContainer" containerID="f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370" Jul 28 13:53:03.369821 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:53:03.369806 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370\": container with ID starting with f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370 not found: ID does not exist" containerID="f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370" Jul 28 13:53:03.369868 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.369825 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370"} err="failed to get container status \"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370\": rpc error: code = NotFound desc = could not find container \"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370\": container with ID starting with f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370 not found: ID does not exist" Jul 28 13:53:03.369868 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.369839 2564 scope.go:117] "RemoveContainer" containerID="b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1" Jul 28 13:53:03.370031 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:53:03.370016 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1\": container with ID starting with b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1 not found: ID does not exist" containerID="b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1" Jul 28 13:53:03.370065 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370033 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1"} err="failed to get container status \"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1\": rpc error: code = NotFound desc = could not find container \"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1\": container with ID starting with b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1 not found: ID does not exist" Jul 28 13:53:03.370065 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370045 2564 scope.go:117] "RemoveContainer" containerID="aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366" Jul 28 13:53:03.370291 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:53:03.370274 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366\": container with ID starting with aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366 not found: ID does not exist" containerID="aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366" Jul 28 13:53:03.370328 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370297 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366"} err="failed to get container status \"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366\": rpc error: code = NotFound desc = could not find container \"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366\": container with ID starting with aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366 not found: ID does not exist" Jul 28 13:53:03.370328 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370313 2564 scope.go:117] "RemoveContainer" containerID="07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22" Jul 28 13:53:03.370534 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:53:03.370517 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22\": container with ID starting with 07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22 not found: ID does not exist" containerID="07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22" Jul 28 13:53:03.370597 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370537 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22"} err="failed to get container status \"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22\": rpc error: code = NotFound desc = could not find container \"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22\": container with ID starting with 07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22 not found: ID does not exist" Jul 28 13:53:03.370597 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370554 2564 scope.go:117] "RemoveContainer" containerID="4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720" Jul 28 13:53:03.370795 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370775 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720"} err="failed to get container status \"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720\": rpc error: code = NotFound desc = could not find container \"4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720\": container with ID starting with 4da3711e921064ba9bb809c4c306fcc96b142ebc26d67db72247ee78f3fe4720 not found: ID does not exist" Jul 28 13:53:03.370863 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370797 2564 scope.go:117] "RemoveContainer" containerID="1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92" Jul 28 13:53:03.371010 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.370994 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92"} err="failed to get container status \"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92\": rpc error: code = NotFound desc = could not find container \"1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92\": container with ID starting with 1fff3832831363c2d6d6e6d7b9a7a26ac98e345790cb17173bf6ef90df38fc92 not found: ID does not exist" Jul 28 13:53:03.371055 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371010 2564 scope.go:117] "RemoveContainer" containerID="f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370" Jul 28 13:53:03.371181 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371164 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370"} err="failed to get container status \"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370\": rpc error: code = NotFound desc = could not find container \"f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370\": container with ID starting with f6fb7dcd463f8b56ac646cbd5057a6e2f7f3a5f55b1c6f729ac1236a752c9370 not found: ID does not exist" Jul 28 13:53:03.371223 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371181 2564 scope.go:117] "RemoveContainer" containerID="b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1" Jul 28 13:53:03.371371 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371354 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1"} err="failed to get container status \"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1\": rpc error: code = NotFound desc = could not find container \"b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1\": container with ID starting with b7cbf726b5140987981b31f21bc6d0af3e10dd121062c771523a94f4bd261bc1 not found: ID does not exist" Jul 28 13:53:03.371434 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371373 2564 scope.go:117] "RemoveContainer" containerID="aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366" Jul 28 13:53:03.371568 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371553 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366"} err="failed to get container status \"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366\": rpc error: code = NotFound desc = could not find container \"aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366\": container with ID starting with aa7ea8d83325c2d7c0a7a5b661b05c5c0814870895ce88ea4d485d59bb461366 not found: ID does not exist" Jul 28 13:53:03.371643 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371568 2564 scope.go:117] "RemoveContainer" containerID="07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22" Jul 28 13:53:03.371799 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:03.371785 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22"} err="failed to get container status \"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22\": rpc error: code = NotFound desc = could not find container \"07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22\": container with ID starting with 07406dddb07f3a751ab725c54e71c6ab076ce2af02721fef2fcfd4b6e20a5a22 not found: ID does not exist" Jul 28 13:53:04.782331 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:04.782298 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" path="/var/lib/kubelet/pods/dc92c226-960c-4314-b6d7-1138a64195a9/volumes" Jul 28 13:53:54.327849 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.327812 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm"] Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328168 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328180 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328193 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="offline" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328200 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="offline" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328206 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328213 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328225 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="feast-init" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328232 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="feast-init" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328241 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="feast-apply" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328247 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="feast-apply" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328253 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="registry" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328258 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="registry" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328323 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="offline" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328332 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="online" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328339 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="ui" Jul 28 13:53:54.328365 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.328346 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc92c226-960c-4314-b6d7-1138a64195a9" containerName="registry" Jul 28 13:53:54.331520 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.331503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.334197 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.334172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 28 13:53:54.334314 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.334233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 28 13:53:54.334314 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.334252 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 28 13:53:54.334314 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.334243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-jljjr\"" Jul 28 13:53:54.334919 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.334894 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 28 13:53:54.335021 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.334986 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 28 13:53:54.335021 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.334996 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 28 13:53:54.335225 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.335207 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 28 13:53:54.346889 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.346870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm"] Jul 28 13:53:54.410427 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.410403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4bac593a-5724-4177-a71b-f8822e38c7ae-registry-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.410506 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.410429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4bac593a-5724-4177-a71b-f8822e38c7ae-feast-data\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.410506 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.410456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6lx\" (UniqueName: \"kubernetes.io/projected/4bac593a-5724-4177-a71b-f8822e38c7ae-kube-api-access-8c6lx\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.410633 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.410553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-offline-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.410633 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.410624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-online-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.410708 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.410653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-ui-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511175 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-offline-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-online-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-ui-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4bac593a-5724-4177-a71b-f8822e38c7ae-registry-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4bac593a-5724-4177-a71b-f8822e38c7ae-feast-data\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511292 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6lx\" (UniqueName: \"kubernetes.io/projected/4bac593a-5724-4177-a71b-f8822e38c7ae-kube-api-access-8c6lx\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511726 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4bac593a-5724-4177-a71b-f8822e38c7ae-feast-data\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.511935 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.511908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4bac593a-5724-4177-a71b-f8822e38c7ae-registry-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.513725 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.513696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-ui-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.513825 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.513806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-offline-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.513884 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.513844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-online-tls\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.519382 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.519360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6lx\" (UniqueName: \"kubernetes.io/projected/4bac593a-5724-4177-a71b-f8822e38c7ae-kube-api-access-8c6lx\") pod \"feast-simple-feast-remote-setup-67f84bfc95-zkmjm\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.641255 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.641202 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:53:54.782765 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:53:54.782730 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bac593a_5724_4177_a71b_f8822e38c7ae.slice/crio-98bd30014e097a9fe6cde3024292880bed5c769b7e983c04aa2c16f1989f2111 WatchSource:0}: Error finding container 98bd30014e097a9fe6cde3024292880bed5c769b7e983c04aa2c16f1989f2111: Status 404 returned error can't find the container with id 98bd30014e097a9fe6cde3024292880bed5c769b7e983c04aa2c16f1989f2111 Jul 28 13:53:54.782869 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:54.782784 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm"] Jul 28 13:53:55.516663 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:55.516616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerStarted","Data":"d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad"} Jul 28 13:53:55.517110 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:55.516679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerStarted","Data":"98bd30014e097a9fe6cde3024292880bed5c769b7e983c04aa2c16f1989f2111"} Jul 28 13:53:59.533503 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:59.533474 2564 generic.go:356] "Generic (PLEG): container finished" podID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerID="d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad" exitCode=0 Jul 28 13:53:59.533839 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:53:59.533520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerDied","Data":"d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad"} Jul 28 13:54:00.540092 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:00.540063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerStarted","Data":"28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490"} Jul 28 13:54:05.563397 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:05.563364 2564 generic.go:356] "Generic (PLEG): container finished" podID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerID="28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490" exitCode=0 Jul 28 13:54:05.563806 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:05.563437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerDied","Data":"28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490"} Jul 28 13:54:06.571977 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.571934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerStarted","Data":"b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86"} Jul 28 13:54:06.572347 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.571998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerStarted","Data":"91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572"} Jul 28 13:54:06.572347 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.572021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerStarted","Data":"fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4"} Jul 28 13:54:06.603110 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.603050 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podStartSLOduration=12.603034536 podStartE2EDuration="12.603034536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:54:06.601714366 +0000 UTC m=+698.303257716" watchObservedRunningTime="2026-07-28 13:54:06.603034536 +0000 UTC m=+698.304577864" Jul 28 13:54:06.641952 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.641913 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:06.642148 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.641983 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:06.642148 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.642012 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:06.643753 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.643713 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Jul 28 13:54:06.643875 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.643730 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 13:54:06.643875 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:06.643730 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 13:54:09.642146 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:09.642108 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Jul 28 13:54:09.642146 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:09.642128 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 13:54:09.642701 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:09.642109 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 13:54:12.643199 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:12.643166 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:12.643712 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:12.643377 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:12.643854 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:12.643826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:12.643973 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:12.643871 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:12.644039 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:12.644002 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:12.644267 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:12.644252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:12.649622 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:12.649599 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:13.603086 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:13.603050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:54:13.607741 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:54:13.607715 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:55:15.690955 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.690918 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm"] Jul 28 13:55:15.691450 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.691201 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" containerID="cri-o://fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4" gracePeriod=30 Jul 28 13:55:15.691450 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.691258 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="offline" containerID="cri-o://91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572" gracePeriod=30 Jul 28 13:55:15.691450 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.691273 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" containerID="cri-o://b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86" gracePeriod=30 Jul 28 13:55:15.790541 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.790505 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2"] Jul 28 13:55:15.790882 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.790797 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="registry" containerID="cri-o://61d82abd6971f057646b7dff15d586ec5fade02f3d6a40498d222501b626ca7a" gracePeriod=30 Jul 28 13:55:15.790882 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.790842 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" containerID="cri-o://386a1b0b528f0d4ae19183e0766926150b836c4908438016dbc184ca1948d1e1" gracePeriod=30 Jul 28 13:55:15.790882 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.790838 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" containerID="cri-o://ba01635eef6b8b2c1d8f225ec3badd3f2f2ee6ea67e72e25abeda95128698467" gracePeriod=30 Jul 28 13:55:15.791141 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:15.790848 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="offline" containerID="cri-o://cdd0b03212d32ab78e8f7c8aa908673be387309011fa27ca7abaaf2c2442e2d9" gracePeriod=30 Jul 28 13:55:16.851308 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.851274 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerID="386a1b0b528f0d4ae19183e0766926150b836c4908438016dbc184ca1948d1e1" exitCode=0 Jul 28 13:55:16.851308 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.851300 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerID="ba01635eef6b8b2c1d8f225ec3badd3f2f2ee6ea67e72e25abeda95128698467" exitCode=0 Jul 28 13:55:16.851800 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.851345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerDied","Data":"386a1b0b528f0d4ae19183e0766926150b836c4908438016dbc184ca1948d1e1"} Jul 28 13:55:16.851800 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.851392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerDied","Data":"ba01635eef6b8b2c1d8f225ec3badd3f2f2ee6ea67e72e25abeda95128698467"} Jul 28 13:55:16.853927 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.853866 2564 generic.go:356] "Generic (PLEG): container finished" podID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerID="b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86" exitCode=0 Jul 28 13:55:16.853927 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.853890 2564 generic.go:356] "Generic (PLEG): container finished" podID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerID="fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4" exitCode=0 Jul 28 13:55:16.854040 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.853940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerDied","Data":"b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86"} Jul 28 13:55:16.854040 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:16.853973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerDied","Data":"fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4"} Jul 28 13:55:20.922406 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:20.922353 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jul 28 13:55:21.268096 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:21.268058 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jul 28 13:55:22.644725 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:22.644680 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 13:55:23.603693 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:23.603649 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 13:55:30.922299 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:30.922254 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jul 28 13:55:31.268134 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:31.268093 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jul 28 13:55:32.644752 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:32.644705 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 13:55:33.603630 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:33.603554 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 13:55:40.921851 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:40.921807 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.23:8443: connect: connection refused" Jul 28 13:55:40.922276 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:40.921945 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:55:41.267737 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:41.267691 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.23:6567/health\": dial tcp 10.133.0.23:6567: connect: connection refused" Jul 28 13:55:41.267890 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:41.267820 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:55:42.643955 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:42.643909 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Jul 28 13:55:42.644431 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:42.644032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:55:43.604276 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:43.604230 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Jul 28 13:55:43.604458 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:43.604366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:55:45.897637 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.897610 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:55:45.924821 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.924790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-online-tls\") pod \"4bac593a-5724-4177-a71b-f8822e38c7ae\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " Jul 28 13:55:45.925019 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.924832 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-offline-tls\") pod \"4bac593a-5724-4177-a71b-f8822e38c7ae\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " Jul 28 13:55:45.925019 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.924863 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-ui-tls\") pod \"4bac593a-5724-4177-a71b-f8822e38c7ae\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " Jul 28 13:55:45.925019 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.924888 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c6lx\" (UniqueName: \"kubernetes.io/projected/4bac593a-5724-4177-a71b-f8822e38c7ae-kube-api-access-8c6lx\") pod \"4bac593a-5724-4177-a71b-f8822e38c7ae\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " Jul 28 13:55:45.925019 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.924911 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4bac593a-5724-4177-a71b-f8822e38c7ae-feast-data\") pod \"4bac593a-5724-4177-a71b-f8822e38c7ae\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " Jul 28 13:55:45.925019 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.924958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4bac593a-5724-4177-a71b-f8822e38c7ae-registry-tls\") pod \"4bac593a-5724-4177-a71b-f8822e38c7ae\" (UID: \"4bac593a-5724-4177-a71b-f8822e38c7ae\") " Jul 28 13:55:45.925532 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.925500 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bac593a-5724-4177-a71b-f8822e38c7ae-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4bac593a-5724-4177-a71b-f8822e38c7ae" (UID: "4bac593a-5724-4177-a71b-f8822e38c7ae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 13:55:45.925685 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.925662 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bac593a-5724-4177-a71b-f8822e38c7ae-feast-data" (OuterVolumeSpecName: "feast-data") pod "4bac593a-5724-4177-a71b-f8822e38c7ae" (UID: "4bac593a-5724-4177-a71b-f8822e38c7ae"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:55:45.927625 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.927594 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-online-tls" (OuterVolumeSpecName: "online-tls") pod "4bac593a-5724-4177-a71b-f8822e38c7ae" (UID: "4bac593a-5724-4177-a71b-f8822e38c7ae"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:55:45.927625 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.927605 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4bac593a-5724-4177-a71b-f8822e38c7ae" (UID: "4bac593a-5724-4177-a71b-f8822e38c7ae"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:55:45.927753 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.927697 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bac593a-5724-4177-a71b-f8822e38c7ae-kube-api-access-8c6lx" (OuterVolumeSpecName: "kube-api-access-8c6lx") pod "4bac593a-5724-4177-a71b-f8822e38c7ae" (UID: "4bac593a-5724-4177-a71b-f8822e38c7ae"). InnerVolumeSpecName "kube-api-access-8c6lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:55:45.927753 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.927728 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4bac593a-5724-4177-a71b-f8822e38c7ae" (UID: "4bac593a-5724-4177-a71b-f8822e38c7ae"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:55:45.968218 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.968127 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerID="cdd0b03212d32ab78e8f7c8aa908673be387309011fa27ca7abaaf2c2442e2d9" exitCode=137 Jul 28 13:55:45.968218 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.968153 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerID="61d82abd6971f057646b7dff15d586ec5fade02f3d6a40498d222501b626ca7a" exitCode=137 Jul 28 13:55:45.968218 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.968199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerDied","Data":"cdd0b03212d32ab78e8f7c8aa908673be387309011fa27ca7abaaf2c2442e2d9"} Jul 28 13:55:45.968430 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.968252 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerDied","Data":"61d82abd6971f057646b7dff15d586ec5fade02f3d6a40498d222501b626ca7a"} Jul 28 13:55:45.970705 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.970681 2564 generic.go:356] "Generic (PLEG): container finished" podID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerID="91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572" exitCode=137 Jul 28 13:55:45.970824 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.970740 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" Jul 28 13:55:45.970824 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.970750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerDied","Data":"91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572"} Jul 28 13:55:45.970824 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.970790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm" event={"ID":"4bac593a-5724-4177-a71b-f8822e38c7ae","Type":"ContainerDied","Data":"98bd30014e097a9fe6cde3024292880bed5c769b7e983c04aa2c16f1989f2111"} Jul 28 13:55:45.970824 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.970814 2564 scope.go:117] "RemoveContainer" containerID="b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86" Jul 28 13:55:45.980329 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.980307 2564 scope.go:117] "RemoveContainer" containerID="91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572" Jul 28 13:55:45.987911 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.987890 2564 scope.go:117] "RemoveContainer" containerID="fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4" Jul 28 13:55:45.994743 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.994718 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm"] Jul 28 13:55:45.995853 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.995837 2564 scope.go:117] "RemoveContainer" containerID="28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490" Jul 28 13:55:45.998552 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:45.998525 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-67f84bfc95-zkmjm"] Jul 28 13:55:46.003786 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.003762 2564 scope.go:117] "RemoveContainer" containerID="d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad" Jul 28 13:55:46.012127 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.012107 2564 scope.go:117] "RemoveContainer" containerID="b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86" Jul 28 13:55:46.012433 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:55:46.012410 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86\": container with ID starting with b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86 not found: ID does not exist" containerID="b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86" Jul 28 13:55:46.012533 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.012440 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86"} err="failed to get container status \"b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86\": rpc error: code = NotFound desc = could not find container \"b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86\": container with ID starting with b026d25a5c32d2216ea178254d1c1737dec5da600d585a770a937d7580227a86 not found: ID does not exist" Jul 28 13:55:46.012533 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.012460 2564 scope.go:117] "RemoveContainer" containerID="91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572" Jul 28 13:55:46.012773 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:55:46.012753 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572\": container with ID starting with 91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572 not found: ID does not exist" containerID="91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572" Jul 28 13:55:46.012816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.012782 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572"} err="failed to get container status \"91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572\": rpc error: code = NotFound desc = could not find container \"91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572\": container with ID starting with 91945ca0cd11a23f5933b1e84856e55faa57e91734f7065c06d906c57a426572 not found: ID does not exist" Jul 28 13:55:46.012816 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.012799 2564 scope.go:117] "RemoveContainer" containerID="fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4" Jul 28 13:55:46.013118 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:55:46.013090 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4\": container with ID starting with fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4 not found: ID does not exist" containerID="fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4" Jul 28 13:55:46.013205 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.013126 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4"} err="failed to get container status \"fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4\": rpc error: code = NotFound desc = could not find container \"fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4\": container with ID starting with fdee2c634f75ac05b93efedbca9a46a3e604a6c0ec3b6acb82e044780d8ceff4 not found: ID does not exist" Jul 28 13:55:46.013205 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.013147 2564 scope.go:117] "RemoveContainer" containerID="28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490" Jul 28 13:55:46.013392 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:55:46.013373 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490\": container with ID starting with 28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490 not found: ID does not exist" containerID="28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490" Jul 28 13:55:46.013438 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.013399 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490"} err="failed to get container status \"28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490\": rpc error: code = NotFound desc = could not find container \"28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490\": container with ID starting with 28d8f06f256d83f94e4ee744be0752d774290f2bd96979bd39487df3d90b7490 not found: ID does not exist" Jul 28 13:55:46.013438 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.013415 2564 scope.go:117] "RemoveContainer" containerID="d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad" Jul 28 13:55:46.013671 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:55:46.013653 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad\": container with ID starting with d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad not found: ID does not exist" containerID="d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad" Jul 28 13:55:46.013719 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.013679 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad"} err="failed to get container status \"d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad\": rpc error: code = NotFound desc = could not find container \"d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad\": container with ID starting with d4b23f116aaa025d12aa110a9fb01aeac4cbce4a736a1105a8153a6dde8e47ad not found: ID does not exist" Jul 28 13:55:46.026059 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.026037 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-online-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.026059 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.026059 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-offline-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.026196 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.026069 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4bac593a-5724-4177-a71b-f8822e38c7ae-ui-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.026196 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.026077 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6lx\" (UniqueName: \"kubernetes.io/projected/4bac593a-5724-4177-a71b-f8822e38c7ae-kube-api-access-8c6lx\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.026196 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.026086 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4bac593a-5724-4177-a71b-f8822e38c7ae-feast-data\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.026196 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.026095 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/4bac593a-5724-4177-a71b-f8822e38c7ae-registry-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.434508 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.434478 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:55:46.529913 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.529873 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2e54e2-15ef-4231-82ad-91e285f248d3-feast-data\") pod \"6e2e54e2-15ef-4231-82ad-91e285f248d3\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " Jul 28 13:55:46.529913 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.529924 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-offline-tls\") pod \"6e2e54e2-15ef-4231-82ad-91e285f248d3\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " Jul 28 13:55:46.530166 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.529944 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-online-tls\") pod \"6e2e54e2-15ef-4231-82ad-91e285f248d3\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " Jul 28 13:55:46.530166 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.529984 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6wpp\" (UniqueName: \"kubernetes.io/projected/6e2e54e2-15ef-4231-82ad-91e285f248d3-kube-api-access-h6wpp\") pod \"6e2e54e2-15ef-4231-82ad-91e285f248d3\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " Jul 28 13:55:46.530166 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.530006 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-ui-tls\") pod \"6e2e54e2-15ef-4231-82ad-91e285f248d3\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " Jul 28 13:55:46.530166 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.530027 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-registry-tls\") pod \"6e2e54e2-15ef-4231-82ad-91e285f248d3\" (UID: \"6e2e54e2-15ef-4231-82ad-91e285f248d3\") " Jul 28 13:55:46.530532 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.530492 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e54e2-15ef-4231-82ad-91e285f248d3-feast-data" (OuterVolumeSpecName: "feast-data") pod "6e2e54e2-15ef-4231-82ad-91e285f248d3" (UID: "6e2e54e2-15ef-4231-82ad-91e285f248d3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:55:46.532273 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.532243 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2e54e2-15ef-4231-82ad-91e285f248d3-kube-api-access-h6wpp" (OuterVolumeSpecName: "kube-api-access-h6wpp") pod "6e2e54e2-15ef-4231-82ad-91e285f248d3" (UID: "6e2e54e2-15ef-4231-82ad-91e285f248d3"). InnerVolumeSpecName "kube-api-access-h6wpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:55:46.532392 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.532264 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6e2e54e2-15ef-4231-82ad-91e285f248d3" (UID: "6e2e54e2-15ef-4231-82ad-91e285f248d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:55:46.532392 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.532274 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6e2e54e2-15ef-4231-82ad-91e285f248d3" (UID: "6e2e54e2-15ef-4231-82ad-91e285f248d3"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:55:46.532513 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.532496 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-online-tls" (OuterVolumeSpecName: "online-tls") pod "6e2e54e2-15ef-4231-82ad-91e285f248d3" (UID: "6e2e54e2-15ef-4231-82ad-91e285f248d3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:55:46.532565 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.532512 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6e2e54e2-15ef-4231-82ad-91e285f248d3" (UID: "6e2e54e2-15ef-4231-82ad-91e285f248d3"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:55:46.631131 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.631039 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-online-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.631131 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.631074 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6wpp\" (UniqueName: \"kubernetes.io/projected/6e2e54e2-15ef-4231-82ad-91e285f248d3-kube-api-access-h6wpp\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.631131 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.631085 2564 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-ui-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.631131 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.631095 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-registry-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.631131 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.631105 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e2e54e2-15ef-4231-82ad-91e285f248d3-feast-data\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.631131 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.631113 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e2e54e2-15ef-4231-82ad-91e285f248d3-offline-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:55:46.782022 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.781989 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" path="/var/lib/kubelet/pods/4bac593a-5724-4177-a71b-f8822e38c7ae/volumes" Jul 28 13:55:46.977879 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.977804 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" Jul 28 13:55:46.978283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.977803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2" event={"ID":"6e2e54e2-15ef-4231-82ad-91e285f248d3","Type":"ContainerDied","Data":"3051f9c02c691f91d66ce99c0eebf256b507e2e526101cb90c3df80299936286"} Jul 28 13:55:46.978283 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.977947 2564 scope.go:117] "RemoveContainer" containerID="386a1b0b528f0d4ae19183e0766926150b836c4908438016dbc184ca1948d1e1" Jul 28 13:55:46.986332 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.986314 2564 scope.go:117] "RemoveContainer" containerID="cdd0b03212d32ab78e8f7c8aa908673be387309011fa27ca7abaaf2c2442e2d9" Jul 28 13:55:46.994016 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:46.993997 2564 scope.go:117] "RemoveContainer" containerID="ba01635eef6b8b2c1d8f225ec3badd3f2f2ee6ea67e72e25abeda95128698467" Jul 28 13:55:47.001129 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:47.001110 2564 scope.go:117] "RemoveContainer" containerID="61d82abd6971f057646b7dff15d586ec5fade02f3d6a40498d222501b626ca7a" Jul 28 13:55:47.005593 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:47.005554 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2"] Jul 28 13:55:47.009206 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:47.009182 2564 scope.go:117] "RemoveContainer" containerID="f682e2300db0e9453c4c77a20268031dedac8223d207e99abcdf7eeeff841955" Jul 28 13:55:47.013135 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:47.013111 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c5f65bd88-xzlg2"] Jul 28 13:55:47.016676 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:47.016658 2564 scope.go:117] "RemoveContainer" containerID="e3900de4ab03005a10b148024e94fdf7f63ddd4bc0146a0d7b310c15e5ee4bde" Jul 28 13:55:48.781569 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:48.781536 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" path="/var/lib/kubelet/pods/6e2e54e2-15ef-4231-82ad-91e285f248d3/volumes" Jul 28 13:55:58.733733 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733646 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-rdm9g"] Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733930 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="feast-init" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733942 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="feast-init" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733955 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="feast-init" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733963 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="feast-init" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733969 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733975 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733982 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="feast-apply" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733988 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="feast-apply" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.733997 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="feast-apply" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734002 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="feast-apply" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734009 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="registry" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734014 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="registry" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734027 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734033 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734037 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734042 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="offline" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734063 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="offline" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734072 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="offline" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734077 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="offline" Jul 28 13:55:58.734073 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734083 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734088 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734130 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="online" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734138 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="offline" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734143 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="registry" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734150 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="offline" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734156 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="ui" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734161 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2e54e2-15ef-4231-82ad-91e285f248d3" containerName="ui" Jul 28 13:55:58.734777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.734166 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bac593a-5724-4177-a71b-f8822e38c7ae" containerName="online" Jul 28 13:55:58.737175 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.737154 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:58.739884 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.739859 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 13:55:58.739884 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.739876 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 28 13:55:58.740088 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.739886 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-pt79r\"" Jul 28 13:55:58.740858 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.740842 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 13:55:58.746057 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.746031 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-rdm9g"] Jul 28 13:55:58.781678 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.781647 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-f6xqw"] Jul 28 13:55:58.784777 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.784759 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-f6xqw" Jul 28 13:55:58.795229 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.795201 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-f6xqw"] Jul 28 13:55:58.827589 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.827537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7ll\" (UniqueName: \"kubernetes.io/projected/c716abdf-6569-4d6e-a4b2-74fd3f84f740-kube-api-access-6l7ll\") pod \"redis-69bbd798b6-f6xqw\" (UID: \"c716abdf-6569-4d6e-a4b2-74fd3f84f740\") " pod="test-ns-feast/redis-69bbd798b6-f6xqw" Jul 28 13:55:58.827793 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.827617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbwt\" (UniqueName: \"kubernetes.io/projected/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-kube-api-access-6gbwt\") pod \"postgres-864846d4f5-rdm9g\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:58.827793 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.827699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-postgresdata\") pod \"postgres-864846d4f5-rdm9g\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:58.928156 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.928126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-postgresdata\") pod \"postgres-864846d4f5-rdm9g\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:58.928332 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.928171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7ll\" (UniqueName: \"kubernetes.io/projected/c716abdf-6569-4d6e-a4b2-74fd3f84f740-kube-api-access-6l7ll\") pod \"redis-69bbd798b6-f6xqw\" (UID: \"c716abdf-6569-4d6e-a4b2-74fd3f84f740\") " pod="test-ns-feast/redis-69bbd798b6-f6xqw" Jul 28 13:55:58.928332 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.928199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbwt\" (UniqueName: \"kubernetes.io/projected/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-kube-api-access-6gbwt\") pod \"postgres-864846d4f5-rdm9g\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:58.928534 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.928515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-postgresdata\") pod \"postgres-864846d4f5-rdm9g\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:58.937425 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.937395 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbwt\" (UniqueName: \"kubernetes.io/projected/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-kube-api-access-6gbwt\") pod \"postgres-864846d4f5-rdm9g\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:58.937557 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:58.937480 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7ll\" (UniqueName: \"kubernetes.io/projected/c716abdf-6569-4d6e-a4b2-74fd3f84f740-kube-api-access-6l7ll\") pod \"redis-69bbd798b6-f6xqw\" (UID: \"c716abdf-6569-4d6e-a4b2-74fd3f84f740\") " pod="test-ns-feast/redis-69bbd798b6-f6xqw" Jul 28 13:55:59.047492 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:59.047388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:55:59.094968 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:59.094935 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-f6xqw" Jul 28 13:55:59.187436 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:59.186545 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-rdm9g"] Jul 28 13:55:59.189441 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:55:59.189413 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ea1925c_8a5c_478b_8bf3_9dffbac408e8.slice/crio-213a7002b63e6d92d4569ae8823a767b38b7d03312403741683e1d8a140cce20 WatchSource:0}: Error finding container 213a7002b63e6d92d4569ae8823a767b38b7d03312403741683e1d8a140cce20: Status 404 returned error can't find the container with id 213a7002b63e6d92d4569ae8823a767b38b7d03312403741683e1d8a140cce20 Jul 28 13:55:59.191258 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:59.191239 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 13:55:59.234923 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:55:59.234894 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-f6xqw"] Jul 28 13:55:59.237348 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:55:59.237321 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc716abdf_6569_4d6e_a4b2_74fd3f84f740.slice/crio-d345a0b7bdf5df1261b3aaa2783be523650bba411421a60b84102d124b28eccb WatchSource:0}: Error finding container d345a0b7bdf5df1261b3aaa2783be523650bba411421a60b84102d124b28eccb: Status 404 returned error can't find the container with id d345a0b7bdf5df1261b3aaa2783be523650bba411421a60b84102d124b28eccb Jul 28 13:56:00.032982 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:00.032947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-rdm9g" event={"ID":"3ea1925c-8a5c-478b-8bf3-9dffbac408e8","Type":"ContainerStarted","Data":"213a7002b63e6d92d4569ae8823a767b38b7d03312403741683e1d8a140cce20"} Jul 28 13:56:00.033949 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:00.033924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-f6xqw" event={"ID":"c716abdf-6569-4d6e-a4b2-74fd3f84f740","Type":"ContainerStarted","Data":"d345a0b7bdf5df1261b3aaa2783be523650bba411421a60b84102d124b28eccb"} Jul 28 13:56:01.042481 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:01.042439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-f6xqw" event={"ID":"c716abdf-6569-4d6e-a4b2-74fd3f84f740","Type":"ContainerStarted","Data":"1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98"} Jul 28 13:56:01.066557 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:01.066480 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-f6xqw" podStartSLOduration=1.73957784 podStartE2EDuration="3.066457983s" totalImagesPullingTime="1.326880143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:56:01.063713843 +0000 UTC m=+812.765257195" watchObservedRunningTime="2026-07-28 13:56:01.066457983 +0000 UTC m=+812.768001334" Jul 28 13:56:05.060775 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:05.060707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-rdm9g" event={"ID":"3ea1925c-8a5c-478b-8bf3-9dffbac408e8","Type":"ContainerStarted","Data":"dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474"} Jul 28 13:56:05.078170 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:05.078101 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-rdm9g" podStartSLOduration=1.627987174 podStartE2EDuration="7.078076758s" totalImagesPullingTime="5.450089584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:56:05.077246396 +0000 UTC m=+816.778789749" watchObservedRunningTime="2026-07-28 13:56:05.078076758 +0000 UTC m=+816.779620087" Jul 28 13:56:09.531877 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.531838 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427"] Jul 28 13:56:09.541809 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.541761 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.603894 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.603853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 28 13:56:09.603894 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.603867 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 28 13:56:09.604152 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.603949 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 28 13:56:09.604152 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.603992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 28 13:56:09.604152 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.604009 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-w8m9k\"" Jul 28 13:56:09.617518 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.617480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-registry-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.617518 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.617518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-offline-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.617669 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.617549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3acb6738-7f4a-4495-8ec6-35595114dd23-feast-data\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.617669 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.617655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-online-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.617732 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.617707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kf9\" (UniqueName: \"kubernetes.io/projected/3acb6738-7f4a-4495-8ec6-35595114dd23-kube-api-access-79kf9\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.639043 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.639009 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427"] Jul 28 13:56:09.718697 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.718662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-online-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.718866 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.718725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kf9\" (UniqueName: \"kubernetes.io/projected/3acb6738-7f4a-4495-8ec6-35595114dd23-kube-api-access-79kf9\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.718866 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.718760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-registry-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.718866 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.718778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-offline-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.718866 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.718806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3acb6738-7f4a-4495-8ec6-35595114dd23-feast-data\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.719266 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.719230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3acb6738-7f4a-4495-8ec6-35595114dd23-feast-data\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.721203 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.721175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-online-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.721324 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.721222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-offline-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.721324 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.721242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-registry-tls\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.728727 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.728698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kf9\" (UniqueName: \"kubernetes.io/projected/3acb6738-7f4a-4495-8ec6-35595114dd23-kube-api-access-79kf9\") pod \"feast-credit-scoring-bb6bc4b66-jl427\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:09.851211 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:09.851107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:10.012165 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:10.012128 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427"] Jul 28 13:56:10.016882 ip-10-0-131-116 kubenswrapper[2564]: W0728 13:56:10.016847 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acb6738_7f4a_4495_8ec6_35595114dd23.slice/crio-b0d1f350dd1bd1b7bdbf726e11c2c2d91d97e455b881c206e148ec1cfd22a2e0 WatchSource:0}: Error finding container b0d1f350dd1bd1b7bdbf726e11c2c2d91d97e455b881c206e148ec1cfd22a2e0: Status 404 returned error can't find the container with id b0d1f350dd1bd1b7bdbf726e11c2c2d91d97e455b881c206e148ec1cfd22a2e0 Jul 28 13:56:10.081378 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:10.081339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerStarted","Data":"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be"} Jul 28 13:56:10.081559 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:10.081411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerStarted","Data":"b0d1f350dd1bd1b7bdbf726e11c2c2d91d97e455b881c206e148ec1cfd22a2e0"} Jul 28 13:56:12.091418 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:12.091385 2564 generic.go:356] "Generic (PLEG): container finished" podID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerID="bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be" exitCode=0 Jul 28 13:56:12.091828 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:12.091447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerDied","Data":"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be"} Jul 28 13:56:13.096230 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:13.096194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerStarted","Data":"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f"} Jul 28 13:56:19.121982 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:19.121949 2564 generic.go:356] "Generic (PLEG): container finished" podID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerID="ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f" exitCode=0 Jul 28 13:56:19.122376 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:19.122024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerDied","Data":"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f"} Jul 28 13:56:20.130877 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:20.130839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerStarted","Data":"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221"} Jul 28 13:56:20.131357 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:20.130920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerStarted","Data":"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab"} Jul 28 13:56:20.131357 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:20.130940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerStarted","Data":"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d"} Jul 28 13:56:20.158330 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:20.158261 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podStartSLOduration=11.158240496 podStartE2EDuration="11.158240496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:56:20.154771572 +0000 UTC m=+831.856314923" watchObservedRunningTime="2026-07-28 13:56:20.158240496 +0000 UTC m=+831.859783825" Jul 28 13:56:21.851589 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:21.851434 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:21.852084 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:21.851651 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:21.852084 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:21.851679 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:21.853255 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:21.853203 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Jul 28 13:56:21.853255 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:21.853244 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 13:56:21.853490 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:21.853277 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Jul 28 13:56:24.852568 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:24.852501 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Jul 28 13:56:24.852568 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:24.852513 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Jul 28 13:56:24.853126 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:24.852516 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 13:56:27.851603 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:27.851545 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Jul 28 13:56:27.852596 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:27.852550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:27.853069 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:27.853047 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:27.853494 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:27.853475 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:27.857519 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:27.857498 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:28.173686 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:28.173607 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:28.176991 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:28.176958 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:29.851589 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:29.851533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:30.852210 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:30.852162 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Jul 28 13:56:33.851870 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:33.851833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:56:33.852730 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:56:33.852699 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:57:57.825225 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.825180 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427"] Jul 28 13:57:57.825993 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.825521 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" containerID="cri-o://65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d" gracePeriod=30 Jul 28 13:57:57.825993 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.825562 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" containerID="cri-o://f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab" gracePeriod=30 Jul 28 13:57:57.825993 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.825599 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="offline" containerID="cri-o://42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221" gracePeriod=30 Jul 28 13:57:57.846556 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.846518 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-rdm9g"] Jul 28 13:57:57.846806 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.846760 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-rdm9g" podUID="3ea1925c-8a5c-478b-8bf3-9dffbac408e8" containerName="postgres" containerID="cri-o://dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474" gracePeriod=30 Jul 28 13:57:57.855814 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.855771 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-f6xqw"] Jul 28 13:57:57.856006 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:57.855981 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-f6xqw" podUID="c716abdf-6569-4d6e-a4b2-74fd3f84f740" containerName="redis" containerID="cri-o://1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98" gracePeriod=30 Jul 28 13:57:58.126001 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.125970 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:57:58.128667 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.128647 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-f6xqw" Jul 28 13:57:58.174275 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.174232 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 13:57:58.211039 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.211004 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l7ll\" (UniqueName: \"kubernetes.io/projected/c716abdf-6569-4d6e-a4b2-74fd3f84f740-kube-api-access-6l7ll\") pod \"c716abdf-6569-4d6e-a4b2-74fd3f84f740\" (UID: \"c716abdf-6569-4d6e-a4b2-74fd3f84f740\") " Jul 28 13:57:58.211238 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.211096 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-postgresdata\") pod \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " Jul 28 13:57:58.211238 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.211152 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gbwt\" (UniqueName: \"kubernetes.io/projected/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-kube-api-access-6gbwt\") pod \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\" (UID: \"3ea1925c-8a5c-478b-8bf3-9dffbac408e8\") " Jul 28 13:57:58.213289 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.213250 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c716abdf-6569-4d6e-a4b2-74fd3f84f740-kube-api-access-6l7ll" (OuterVolumeSpecName: "kube-api-access-6l7ll") pod "c716abdf-6569-4d6e-a4b2-74fd3f84f740" (UID: "c716abdf-6569-4d6e-a4b2-74fd3f84f740"). InnerVolumeSpecName "kube-api-access-6l7ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:57:58.213405 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.213288 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-kube-api-access-6gbwt" (OuterVolumeSpecName: "kube-api-access-6gbwt") pod "3ea1925c-8a5c-478b-8bf3-9dffbac408e8" (UID: "3ea1925c-8a5c-478b-8bf3-9dffbac408e8"). InnerVolumeSpecName "kube-api-access-6gbwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:57:58.241053 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.241008 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "3ea1925c-8a5c-478b-8bf3-9dffbac408e8" (UID: "3ea1925c-8a5c-478b-8bf3-9dffbac408e8"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:57:58.312052 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.312016 2564 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-postgresdata\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:57:58.312052 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.312045 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gbwt\" (UniqueName: \"kubernetes.io/projected/3ea1925c-8a5c-478b-8bf3-9dffbac408e8-kube-api-access-6gbwt\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:57:58.312052 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.312056 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l7ll\" (UniqueName: \"kubernetes.io/projected/c716abdf-6569-4d6e-a4b2-74fd3f84f740-kube-api-access-6l7ll\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:57:58.530492 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.530458 2564 generic.go:356] "Generic (PLEG): container finished" podID="c716abdf-6569-4d6e-a4b2-74fd3f84f740" containerID="1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98" exitCode=0 Jul 28 13:57:58.530675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.530501 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-f6xqw" Jul 28 13:57:58.530675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.530542 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-f6xqw" event={"ID":"c716abdf-6569-4d6e-a4b2-74fd3f84f740","Type":"ContainerDied","Data":"1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98"} Jul 28 13:57:58.530675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.530602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-f6xqw" event={"ID":"c716abdf-6569-4d6e-a4b2-74fd3f84f740","Type":"ContainerDied","Data":"d345a0b7bdf5df1261b3aaa2783be523650bba411421a60b84102d124b28eccb"} Jul 28 13:57:58.530675 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.530620 2564 scope.go:117] "RemoveContainer" containerID="1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98" Jul 28 13:57:58.531972 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.531947 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ea1925c-8a5c-478b-8bf3-9dffbac408e8" containerID="dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474" exitCode=0 Jul 28 13:57:58.532087 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.531989 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-rdm9g" Jul 28 13:57:58.532087 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.532021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-rdm9g" event={"ID":"3ea1925c-8a5c-478b-8bf3-9dffbac408e8","Type":"ContainerDied","Data":"dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474"} Jul 28 13:57:58.532087 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.532050 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-rdm9g" event={"ID":"3ea1925c-8a5c-478b-8bf3-9dffbac408e8","Type":"ContainerDied","Data":"213a7002b63e6d92d4569ae8823a767b38b7d03312403741683e1d8a140cce20"} Jul 28 13:57:58.539976 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.539958 2564 scope.go:117] "RemoveContainer" containerID="1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98" Jul 28 13:57:58.540305 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:57:58.540268 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98\": container with ID starting with 1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98 not found: ID does not exist" containerID="1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98" Jul 28 13:57:58.540409 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.540330 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98"} err="failed to get container status \"1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98\": rpc error: code = NotFound desc = could not find container \"1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98\": container with ID starting with 1a1f4779418d5fb78efcce44bc87d933391fc298696e5cb3935b84a1096a3d98 not found: ID does not exist" Jul 28 13:57:58.540409 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.540356 2564 scope.go:117] "RemoveContainer" containerID="dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474" Jul 28 13:57:58.549649 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.549627 2564 scope.go:117] "RemoveContainer" containerID="dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474" Jul 28 13:57:58.549946 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:57:58.549926 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474\": container with ID starting with dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474 not found: ID does not exist" containerID="dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474" Jul 28 13:57:58.550011 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.549959 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474"} err="failed to get container status \"dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474\": rpc error: code = NotFound desc = could not find container \"dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474\": container with ID starting with dd48cd36b2c010251531b5ee9739406d1004446797a48871f81cc89b9a546474 not found: ID does not exist" Jul 28 13:57:58.555366 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.555338 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-rdm9g"] Jul 28 13:57:58.560991 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.560967 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-rdm9g"] Jul 28 13:57:58.572197 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.572167 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-f6xqw"] Jul 28 13:57:58.575558 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.575535 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-f6xqw"] Jul 28 13:57:58.782952 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.782865 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea1925c-8a5c-478b-8bf3-9dffbac408e8" path="/var/lib/kubelet/pods/3ea1925c-8a5c-478b-8bf3-9dffbac408e8/volumes" Jul 28 13:57:58.783187 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:58.783174 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c716abdf-6569-4d6e-a4b2-74fd3f84f740" path="/var/lib/kubelet/pods/c716abdf-6569-4d6e-a4b2-74fd3f84f740/volumes" Jul 28 13:57:59.539729 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:59.539696 2564 generic.go:356] "Generic (PLEG): container finished" podID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerID="f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab" exitCode=0 Jul 28 13:57:59.540112 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:57:59.539766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerDied","Data":"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab"} Jul 28 13:58:08.173888 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:08.173837 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 13:58:18.174210 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:18.174170 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 13:58:18.174630 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:18.174286 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:58:27.854222 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:27.854169 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Jul 28 13:58:28.174366 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.174273 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Jul 28 13:58:28.483286 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.483256 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:58:28.528277 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.528244 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79kf9\" (UniqueName: \"kubernetes.io/projected/3acb6738-7f4a-4495-8ec6-35595114dd23-kube-api-access-79kf9\") pod \"3acb6738-7f4a-4495-8ec6-35595114dd23\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " Jul 28 13:58:28.528468 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.528295 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-online-tls\") pod \"3acb6738-7f4a-4495-8ec6-35595114dd23\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " Jul 28 13:58:28.528468 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.528336 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3acb6738-7f4a-4495-8ec6-35595114dd23-feast-data\") pod \"3acb6738-7f4a-4495-8ec6-35595114dd23\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " Jul 28 13:58:28.528468 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.528389 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-offline-tls\") pod \"3acb6738-7f4a-4495-8ec6-35595114dd23\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " Jul 28 13:58:28.528468 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.528462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-registry-tls\") pod \"3acb6738-7f4a-4495-8ec6-35595114dd23\" (UID: \"3acb6738-7f4a-4495-8ec6-35595114dd23\") " Jul 28 13:58:28.530554 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.530520 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-online-tls" (OuterVolumeSpecName: "online-tls") pod "3acb6738-7f4a-4495-8ec6-35595114dd23" (UID: "3acb6738-7f4a-4495-8ec6-35595114dd23"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:58:28.530673 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.530600 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3acb6738-7f4a-4495-8ec6-35595114dd23" (UID: "3acb6738-7f4a-4495-8ec6-35595114dd23"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:58:28.530867 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.530841 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "3acb6738-7f4a-4495-8ec6-35595114dd23" (UID: "3acb6738-7f4a-4495-8ec6-35595114dd23"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 13:58:28.530923 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.530853 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acb6738-7f4a-4495-8ec6-35595114dd23-kube-api-access-79kf9" (OuterVolumeSpecName: "kube-api-access-79kf9") pod "3acb6738-7f4a-4495-8ec6-35595114dd23" (UID: "3acb6738-7f4a-4495-8ec6-35595114dd23"). InnerVolumeSpecName "kube-api-access-79kf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 13:58:28.536920 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.536891 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acb6738-7f4a-4495-8ec6-35595114dd23-feast-data" (OuterVolumeSpecName: "feast-data") pod "3acb6738-7f4a-4495-8ec6-35595114dd23" (UID: "3acb6738-7f4a-4495-8ec6-35595114dd23"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 13:58:28.629425 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.629387 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-registry-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:58:28.629425 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.629420 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kf9\" (UniqueName: \"kubernetes.io/projected/3acb6738-7f4a-4495-8ec6-35595114dd23-kube-api-access-79kf9\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:58:28.629425 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.629433 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-online-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:58:28.629680 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.629441 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3acb6738-7f4a-4495-8ec6-35595114dd23-feast-data\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:58:28.629680 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.629449 2564 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3acb6738-7f4a-4495-8ec6-35595114dd23-offline-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 13:58:28.659003 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.658968 2564 generic.go:356] "Generic (PLEG): container finished" podID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerID="42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221" exitCode=137 Jul 28 13:58:28.659003 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.659001 2564 generic.go:356] "Generic (PLEG): container finished" podID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerID="65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d" exitCode=137 Jul 28 13:58:28.659175 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.659040 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" Jul 28 13:58:28.659175 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.659038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerDied","Data":"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221"} Jul 28 13:58:28.659175 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.659095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerDied","Data":"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d"} Jul 28 13:58:28.659175 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.659113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427" event={"ID":"3acb6738-7f4a-4495-8ec6-35595114dd23","Type":"ContainerDied","Data":"b0d1f350dd1bd1b7bdbf726e11c2c2d91d97e455b881c206e148ec1cfd22a2e0"} Jul 28 13:58:28.659175 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.659138 2564 scope.go:117] "RemoveContainer" containerID="42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221" Jul 28 13:58:28.667681 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.667652 2564 scope.go:117] "RemoveContainer" containerID="f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab" Jul 28 13:58:28.675284 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.675265 2564 scope.go:117] "RemoveContainer" containerID="65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d" Jul 28 13:58:28.683403 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.683382 2564 scope.go:117] "RemoveContainer" containerID="ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f" Jul 28 13:58:28.684635 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.684616 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427"] Jul 28 13:58:28.690690 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.690664 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-bb6bc4b66-jl427"] Jul 28 13:58:28.691159 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.691145 2564 scope.go:117] "RemoveContainer" containerID="bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be" Jul 28 13:58:28.698442 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.698425 2564 scope.go:117] "RemoveContainer" containerID="42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221" Jul 28 13:58:28.698762 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:58:28.698741 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221\": container with ID starting with 42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221 not found: ID does not exist" containerID="42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221" Jul 28 13:58:28.698818 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.698771 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221"} err="failed to get container status \"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221\": rpc error: code = NotFound desc = could not find container \"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221\": container with ID starting with 42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221 not found: ID does not exist" Jul 28 13:58:28.698818 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.698792 2564 scope.go:117] "RemoveContainer" containerID="f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab" Jul 28 13:58:28.699018 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:58:28.699003 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab\": container with ID starting with f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab not found: ID does not exist" containerID="f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab" Jul 28 13:58:28.699061 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699023 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab"} err="failed to get container status \"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab\": rpc error: code = NotFound desc = could not find container \"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab\": container with ID starting with f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab not found: ID does not exist" Jul 28 13:58:28.699061 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699037 2564 scope.go:117] "RemoveContainer" containerID="65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d" Jul 28 13:58:28.699304 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:58:28.699286 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d\": container with ID starting with 65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d not found: ID does not exist" containerID="65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d" Jul 28 13:58:28.699356 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699309 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d"} err="failed to get container status \"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d\": rpc error: code = NotFound desc = could not find container \"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d\": container with ID starting with 65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d not found: ID does not exist" Jul 28 13:58:28.699356 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699324 2564 scope.go:117] "RemoveContainer" containerID="ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f" Jul 28 13:58:28.699545 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:58:28.699529 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f\": container with ID starting with ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f not found: ID does not exist" containerID="ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f" Jul 28 13:58:28.699662 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699549 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f"} err="failed to get container status \"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f\": rpc error: code = NotFound desc = could not find container \"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f\": container with ID starting with ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f not found: ID does not exist" Jul 28 13:58:28.699662 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699561 2564 scope.go:117] "RemoveContainer" containerID="bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be" Jul 28 13:58:28.699839 ip-10-0-131-116 kubenswrapper[2564]: E0728 13:58:28.699822 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be\": container with ID starting with bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be not found: ID does not exist" containerID="bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be" Jul 28 13:58:28.699879 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699843 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be"} err="failed to get container status \"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be\": rpc error: code = NotFound desc = could not find container \"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be\": container with ID starting with bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be not found: ID does not exist" Jul 28 13:58:28.699879 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.699858 2564 scope.go:117] "RemoveContainer" containerID="42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221" Jul 28 13:58:28.700090 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700062 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221"} err="failed to get container status \"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221\": rpc error: code = NotFound desc = could not find container \"42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221\": container with ID starting with 42b17bbf885d0cdcbf37657b83d467a81903acbe68355d2e6bba86c22eb33221 not found: ID does not exist" Jul 28 13:58:28.700131 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700090 2564 scope.go:117] "RemoveContainer" containerID="f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab" Jul 28 13:58:28.700311 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700295 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab"} err="failed to get container status \"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab\": rpc error: code = NotFound desc = could not find container \"f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab\": container with ID starting with f746489ccca97e902ffc03a8daaa7c19cd962ce4ae06ba6c08efc21e0704ffab not found: ID does not exist" Jul 28 13:58:28.700354 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700311 2564 scope.go:117] "RemoveContainer" containerID="65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d" Jul 28 13:58:28.700509 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700493 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d"} err="failed to get container status \"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d\": rpc error: code = NotFound desc = could not find container \"65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d\": container with ID starting with 65c24e61046df1f76cb2d046c7370f94ef16a8d0e248e1ef1cb7e9e566a6562d not found: ID does not exist" Jul 28 13:58:28.700569 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700509 2564 scope.go:117] "RemoveContainer" containerID="ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f" Jul 28 13:58:28.700748 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700730 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f"} err="failed to get container status \"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f\": rpc error: code = NotFound desc = could not find container \"ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f\": container with ID starting with ce48c248228752a6aa990abdcb0a9adbb4db2b1481e934238be77ee9ee884f1f not found: ID does not exist" Jul 28 13:58:28.700803 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700747 2564 scope.go:117] "RemoveContainer" containerID="bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be" Jul 28 13:58:28.700945 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.700929 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be"} err="failed to get container status \"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be\": rpc error: code = NotFound desc = could not find container \"bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be\": container with ID starting with bd0b00cb704cfa3723681135e97854f0f684fd9055d283ca1ce535d3525f77be not found: ID does not exist" Jul 28 13:58:28.782159 ip-10-0-131-116 kubenswrapper[2564]: I0728 13:58:28.782049 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" path="/var/lib/kubelet/pods/3acb6738-7f4a-4495-8ec6-35595114dd23/volumes" Jul 28 14:02:23.378341 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378307 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d9ndd"] Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378605 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c716abdf-6569-4d6e-a4b2-74fd3f84f740" containerName="redis" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378617 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c716abdf-6569-4d6e-a4b2-74fd3f84f740" containerName="redis" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378626 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378632 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378641 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="offline" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378647 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="offline" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378655 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea1925c-8a5c-478b-8bf3-9dffbac408e8" containerName="postgres" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378660 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea1925c-8a5c-478b-8bf3-9dffbac408e8" containerName="postgres" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378666 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="feast-apply" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378673 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="feast-apply" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378681 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378685 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378697 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="feast-init" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378701 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="feast-init" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378748 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="registry" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378756 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="online" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378763 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acb6738-7f4a-4495-8ec6-35595114dd23" containerName="offline" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378769 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea1925c-8a5c-478b-8bf3-9dffbac408e8" containerName="postgres" Jul 28 14:02:23.378824 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.378774 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c716abdf-6569-4d6e-a4b2-74fd3f84f740" containerName="redis" Jul 28 14:02:23.381447 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.381424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" Jul 28 14:02:23.383855 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.383812 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 28 14:02:23.383855 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.383825 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-x5pq2\"" Jul 28 14:02:23.383855 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.383857 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 28 14:02:23.397620 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.397569 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d9ndd"] Jul 28 14:02:23.441288 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.441249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5nlz\" (UniqueName: \"kubernetes.io/projected/174e602e-e8de-414a-aa5d-ce6f5ac47700-kube-api-access-c5nlz\") pod \"redis-69bbd798b6-d9ndd\" (UID: \"174e602e-e8de-414a-aa5d-ce6f5ac47700\") " pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" Jul 28 14:02:23.542285 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.542247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5nlz\" (UniqueName: \"kubernetes.io/projected/174e602e-e8de-414a-aa5d-ce6f5ac47700-kube-api-access-c5nlz\") pod \"redis-69bbd798b6-d9ndd\" (UID: \"174e602e-e8de-414a-aa5d-ce6f5ac47700\") " pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" Jul 28 14:02:23.551251 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.551217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5nlz\" (UniqueName: \"kubernetes.io/projected/174e602e-e8de-414a-aa5d-ce6f5ac47700-kube-api-access-c5nlz\") pod \"redis-69bbd798b6-d9ndd\" (UID: \"174e602e-e8de-414a-aa5d-ce6f5ac47700\") " pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" Jul 28 14:02:23.690510 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.690412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" Jul 28 14:02:23.825758 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:23.825455 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d9ndd"] Jul 28 14:02:23.828007 ip-10-0-131-116 kubenswrapper[2564]: W0728 14:02:23.827974 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod174e602e_e8de_414a_aa5d_ce6f5ac47700.slice/crio-f07c99e7d1997005bf465ff905bee94283f5970982fd29fd1d2eec373d521291 WatchSource:0}: Error finding container f07c99e7d1997005bf465ff905bee94283f5970982fd29fd1d2eec373d521291: Status 404 returned error can't find the container with id f07c99e7d1997005bf465ff905bee94283f5970982fd29fd1d2eec373d521291 Jul 28 14:02:24.516727 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:24.516690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" event={"ID":"174e602e-e8de-414a-aa5d-ce6f5ac47700","Type":"ContainerStarted","Data":"9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79"} Jul 28 14:02:24.517123 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:24.516738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" event={"ID":"174e602e-e8de-414a-aa5d-ce6f5ac47700","Type":"ContainerStarted","Data":"f07c99e7d1997005bf465ff905bee94283f5970982fd29fd1d2eec373d521291"} Jul 28 14:02:24.534080 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:24.534028 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" podStartSLOduration=1.534010109 podStartE2EDuration="1.534010109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:02:24.53295528 +0000 UTC m=+1196.234498609" watchObservedRunningTime="2026-07-28 14:02:24.534010109 +0000 UTC m=+1196.235553438" Jul 28 14:02:28.807046 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.807008 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-qgtlm"] Jul 28 14:02:28.810740 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.810703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:28.813597 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.813550 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 28 14:02:28.817956 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.817927 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-qgtlm"] Jul 28 14:02:28.884228 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.884186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc29\" (UniqueName: \"kubernetes.io/projected/ee123a15-39e9-4a1b-99fc-0d6940096f92-kube-api-access-dhc29\") pod \"postgres-864846d4f5-qgtlm\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:28.884228 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.884234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee123a15-39e9-4a1b-99fc-0d6940096f92-postgresdata\") pod \"postgres-864846d4f5-qgtlm\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:28.985259 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.985217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee123a15-39e9-4a1b-99fc-0d6940096f92-postgresdata\") pod \"postgres-864846d4f5-qgtlm\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:28.985441 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.985320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc29\" (UniqueName: \"kubernetes.io/projected/ee123a15-39e9-4a1b-99fc-0d6940096f92-kube-api-access-dhc29\") pod \"postgres-864846d4f5-qgtlm\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:28.985616 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.985594 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee123a15-39e9-4a1b-99fc-0d6940096f92-postgresdata\") pod \"postgres-864846d4f5-qgtlm\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:28.996790 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:28.996762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc29\" (UniqueName: \"kubernetes.io/projected/ee123a15-39e9-4a1b-99fc-0d6940096f92-kube-api-access-dhc29\") pod \"postgres-864846d4f5-qgtlm\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:29.121249 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:29.121162 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:02:29.253771 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:29.253735 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-qgtlm"] Jul 28 14:02:29.256946 ip-10-0-131-116 kubenswrapper[2564]: W0728 14:02:29.256916 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee123a15_39e9_4a1b_99fc_0d6940096f92.slice/crio-34fc622c674807ec6ddbbf1eb055042c29363538b31b99cd3a2302ec29eed72f WatchSource:0}: Error finding container 34fc622c674807ec6ddbbf1eb055042c29363538b31b99cd3a2302ec29eed72f: Status 404 returned error can't find the container with id 34fc622c674807ec6ddbbf1eb055042c29363538b31b99cd3a2302ec29eed72f Jul 28 14:02:29.535841 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:29.535808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" event={"ID":"ee123a15-39e9-4a1b-99fc-0d6940096f92","Type":"ContainerStarted","Data":"aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd"} Jul 28 14:02:29.536019 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:29.535851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" event={"ID":"ee123a15-39e9-4a1b-99fc-0d6940096f92","Type":"ContainerStarted","Data":"34fc622c674807ec6ddbbf1eb055042c29363538b31b99cd3a2302ec29eed72f"} Jul 28 14:02:29.555408 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:29.555339 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" podStartSLOduration=1.555316837 podStartE2EDuration="1.555316837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:02:29.553414603 +0000 UTC m=+1201.254957928" watchObservedRunningTime="2026-07-28 14:02:29.555316837 +0000 UTC m=+1201.256860166" Jul 28 14:02:34.421365 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.421335 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg"] Jul 28 14:02:34.424764 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.424744 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.427460 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.427430 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 28 14:02:34.427608 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.427481 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 28 14:02:34.427608 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.427513 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 28 14:02:34.427608 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.427486 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-8l28j\"" Jul 28 14:02:34.439653 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.439619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg"] Jul 28 14:02:34.527930 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.527890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.527930 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.527931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/784a33fc-498b-4ece-8c59-6c299cf62651-feast-data\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.528191 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.527983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-registry-tls\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.528191 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.528042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mhp\" (UniqueName: \"kubernetes.io/projected/784a33fc-498b-4ece-8c59-6c299cf62651-kube-api-access-l9mhp\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.629497 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.629456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.629716 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.629510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/784a33fc-498b-4ece-8c59-6c299cf62651-feast-data\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.629716 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.629603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-registry-tls\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.629716 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:02:34.629633 2564 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Jul 28 14:02:34.629716 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:02:34.629702 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls podName:784a33fc-498b-4ece-8c59-6c299cf62651 nodeName:}" failed. No retries permitted until 2026-07-28 14:02:35.129685453 +0000 UTC m=+1206.831228758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls") pod "feast-credit-scoring-5c6f98c969-xpfkg" (UID: "784a33fc-498b-4ece-8c59-6c299cf62651") : secret "feast-credit-scoring-online-tls" not found Jul 28 14:02:34.629893 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.629633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mhp\" (UniqueName: \"kubernetes.io/projected/784a33fc-498b-4ece-8c59-6c299cf62651-kube-api-access-l9mhp\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.629944 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.629923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/784a33fc-498b-4ece-8c59-6c299cf62651-feast-data\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.632063 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.632038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-registry-tls\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:34.641003 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:34.640973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mhp\" (UniqueName: \"kubernetes.io/projected/784a33fc-498b-4ece-8c59-6c299cf62651-kube-api-access-l9mhp\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:35.133292 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:35.133251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:35.135635 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:35.135606 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls\") pod \"feast-credit-scoring-5c6f98c969-xpfkg\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:35.333767 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:35.333720 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:35.479240 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:35.479212 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg"] Jul 28 14:02:35.479869 ip-10-0-131-116 kubenswrapper[2564]: W0728 14:02:35.479838 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784a33fc_498b_4ece_8c59_6c299cf62651.slice/crio-6634640a12449a9b59ee4934892ed8859a6324a6ce5fcbdc93c266e18e565d1b WatchSource:0}: Error finding container 6634640a12449a9b59ee4934892ed8859a6324a6ce5fcbdc93c266e18e565d1b: Status 404 returned error can't find the container with id 6634640a12449a9b59ee4934892ed8859a6324a6ce5fcbdc93c266e18e565d1b Jul 28 14:02:35.561120 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:35.561081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerStarted","Data":"b6085c0b857c471c82754cf6996d01dad9ade29e9f449f75acb60b86f17d12e0"} Jul 28 14:02:35.561120 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:35.561147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerStarted","Data":"6634640a12449a9b59ee4934892ed8859a6324a6ce5fcbdc93c266e18e565d1b"} Jul 28 14:02:36.566419 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:36.566387 2564 generic.go:356] "Generic (PLEG): container finished" podID="784a33fc-498b-4ece-8c59-6c299cf62651" containerID="b6085c0b857c471c82754cf6996d01dad9ade29e9f449f75acb60b86f17d12e0" exitCode=0 Jul 28 14:02:36.566826 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:36.566473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerDied","Data":"b6085c0b857c471c82754cf6996d01dad9ade29e9f449f75acb60b86f17d12e0"} Jul 28 14:02:37.573039 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:37.573008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerStarted","Data":"d7faab0556e00e2d3fb47acc6cf7fc5cae5d79f1739d9642f8b8accf313af257"} Jul 28 14:02:44.602300 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:44.602260 2564 generic.go:356] "Generic (PLEG): container finished" podID="784a33fc-498b-4ece-8c59-6c299cf62651" containerID="d7faab0556e00e2d3fb47acc6cf7fc5cae5d79f1739d9642f8b8accf313af257" exitCode=0 Jul 28 14:02:44.602707 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:44.602334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerDied","Data":"d7faab0556e00e2d3fb47acc6cf7fc5cae5d79f1739d9642f8b8accf313af257"} Jul 28 14:02:45.609624 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:45.609570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerStarted","Data":"4a6d050ba8bb445bd7bf329d44007a49b7ec40c8cc674051821d65d32125a0a2"} Jul 28 14:02:45.610084 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:45.609654 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerStarted","Data":"10f5e0b09f1402a31af976dbb31c1eedb0ed95b729524d3bf7902956964d066e"} Jul 28 14:02:45.639254 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:45.639193 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podStartSLOduration=11.639177287999999 podStartE2EDuration="11.639177288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:02:45.638931004 +0000 UTC m=+1217.340474333" watchObservedRunningTime="2026-07-28 14:02:45.639177288 +0000 UTC m=+1217.340720615" Jul 28 14:02:47.334054 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:47.334007 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:47.334433 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:47.334089 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:47.335671 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:47.335642 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jul 28 14:02:47.335728 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:47.335676 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 28 14:02:50.334602 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:50.334549 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jul 28 14:02:50.335030 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:50.334675 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 28 14:02:53.334277 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:53.334226 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused" Jul 28 14:02:53.339864 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:53.339836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:53.340177 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:53.340151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:53.343173 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:53.343143 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:55.333890 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:55.333844 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:56.334768 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:56.334738 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:02:56.657937 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:02:56.657858 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:03:00.376111 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.376074 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c"] Jul 28 14:03:00.379713 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.379695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.383243 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.383222 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 28 14:03:00.383648 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.383623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-5pmdv\"" Jul 28 14:03:00.383757 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.383652 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 28 14:03:00.384200 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.384185 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 28 14:03:00.395076 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.395051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c"] Jul 28 14:03:00.449547 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.449512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-online-tls\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.449547 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.449545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c069cc-9528-4fb5-ad4e-c551d751af54-feast-data\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.449811 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.449567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26f6p\" (UniqueName: \"kubernetes.io/projected/b2c069cc-9528-4fb5-ad4e-c551d751af54-kube-api-access-26f6p\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.449811 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.449686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-registry-tls\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.550198 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.550161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-online-tls\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.550198 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.550197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c069cc-9528-4fb5-ad4e-c551d751af54-feast-data\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.550442 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.550217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26f6p\" (UniqueName: \"kubernetes.io/projected/b2c069cc-9528-4fb5-ad4e-c551d751af54-kube-api-access-26f6p\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.550442 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.550274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-registry-tls\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.550642 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.550605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c069cc-9528-4fb5-ad4e-c551d751af54-feast-data\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.552748 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.552717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-registry-tls\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.552840 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.552777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-online-tls\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.559166 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.559145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26f6p\" (UniqueName: \"kubernetes.io/projected/b2c069cc-9528-4fb5-ad4e-c551d751af54-kube-api-access-26f6p\") pod \"feast-driver-ranking-6479d5dc78-kxj2c\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.689913 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.689820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:00.837336 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:00.837311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c"] Jul 28 14:03:00.839421 ip-10-0-131-116 kubenswrapper[2564]: W0728 14:03:00.839384 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c069cc_9528_4fb5_ad4e_c551d751af54.slice/crio-25d6940ed48bb9ec6125a997b06126999afe7178111a0a8bb0e442d8e80e7954 WatchSource:0}: Error finding container 25d6940ed48bb9ec6125a997b06126999afe7178111a0a8bb0e442d8e80e7954: Status 404 returned error can't find the container with id 25d6940ed48bb9ec6125a997b06126999afe7178111a0a8bb0e442d8e80e7954 Jul 28 14:03:01.676386 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:01.676347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerStarted","Data":"0620ffda6181f1d97a69b4035db86eaaa73b6a7900bdf97b2424306b392231fe"} Jul 28 14:03:01.676831 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:01.676402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerStarted","Data":"25d6940ed48bb9ec6125a997b06126999afe7178111a0a8bb0e442d8e80e7954"} Jul 28 14:03:06.698281 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:06.698246 2564 generic.go:356] "Generic (PLEG): container finished" podID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerID="0620ffda6181f1d97a69b4035db86eaaa73b6a7900bdf97b2424306b392231fe" exitCode=0 Jul 28 14:03:06.698781 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:06.698322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerDied","Data":"0620ffda6181f1d97a69b4035db86eaaa73b6a7900bdf97b2424306b392231fe"} Jul 28 14:03:07.703693 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:07.703658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerStarted","Data":"5b826fd2ee625c388b7b3854702d5cfb6335454f9b81b4ece47ddcbeb0367f46"} Jul 28 14:03:14.735914 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:14.735878 2564 generic.go:356] "Generic (PLEG): container finished" podID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerID="5b826fd2ee625c388b7b3854702d5cfb6335454f9b81b4ece47ddcbeb0367f46" exitCode=0 Jul 28 14:03:14.736297 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:14.735955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerDied","Data":"5b826fd2ee625c388b7b3854702d5cfb6335454f9b81b4ece47ddcbeb0367f46"} Jul 28 14:03:15.744112 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:15.744079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerStarted","Data":"833dbdcf91f678e4167c9236232fec05508af3309460d0c36ab28dc98a68fba1"} Jul 28 14:03:15.744513 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:15.744136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerStarted","Data":"977a58499e56ceb20c23115cc07e7da80e984e08b0797fe2a99cf18f4997ad79"} Jul 28 14:03:15.776107 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:15.776046 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podStartSLOduration=15.776026052 podStartE2EDuration="15.776026052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:03:15.775531887 +0000 UTC m=+1247.477075243" watchObservedRunningTime="2026-07-28 14:03:15.776026052 +0000 UTC m=+1247.477569383" Jul 28 14:03:18.690310 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:18.690271 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:18.690815 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:18.690352 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:18.691962 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:18.691923 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jul 28 14:03:18.691962 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:18.691942 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 28 14:03:20.690556 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:20.690509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:20.690937 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:20.690618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:21.691326 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:21.691232 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jul 28 14:03:21.697385 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:21.697356 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:21.701520 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:21.701480 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:24.690630 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:24.690559 2564 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" Jul 28 14:03:27.691178 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:27.691147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:03:27.691940 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:03:27.691921 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:04:46.939549 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:46.939513 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg"] Jul 28 14:04:46.940165 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:46.939810 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="registry" containerID="cri-o://10f5e0b09f1402a31af976dbb31c1eedb0ed95b729524d3bf7902956964d066e" gracePeriod=30 Jul 28 14:04:46.940165 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:46.939912 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" containerID="cri-o://4a6d050ba8bb445bd7bf329d44007a49b7ec40c8cc674051821d65d32125a0a2" gracePeriod=30 Jul 28 14:04:46.958778 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:46.958740 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c"] Jul 28 14:04:46.959384 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:46.959330 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="registry" containerID="cri-o://977a58499e56ceb20c23115cc07e7da80e984e08b0797fe2a99cf18f4997ad79" gracePeriod=30 Jul 28 14:04:46.959760 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:46.959733 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" containerID="cri-o://833dbdcf91f678e4167c9236232fec05508af3309460d0c36ab28dc98a68fba1" gracePeriod=30 Jul 28 14:04:48.137331 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.137296 2564 generic.go:356] "Generic (PLEG): container finished" podID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerID="833dbdcf91f678e4167c9236232fec05508af3309460d0c36ab28dc98a68fba1" exitCode=0 Jul 28 14:04:48.137822 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.137370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerDied","Data":"833dbdcf91f678e4167c9236232fec05508af3309460d0c36ab28dc98a68fba1"} Jul 28 14:04:48.139701 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.139673 2564 generic.go:356] "Generic (PLEG): container finished" podID="784a33fc-498b-4ece-8c59-6c299cf62651" containerID="4a6d050ba8bb445bd7bf329d44007a49b7ec40c8cc674051821d65d32125a0a2" exitCode=0 Jul 28 14:04:48.139823 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.139709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerDied","Data":"4a6d050ba8bb445bd7bf329d44007a49b7ec40c8cc674051821d65d32125a0a2"} Jul 28 14:04:48.668801 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.668771 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-qgtlm"] Jul 28 14:04:48.668977 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.668956 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" podUID="ee123a15-39e9-4a1b-99fc-0d6940096f92" containerName="postgres" containerID="cri-o://aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd" gracePeriod=30 Jul 28 14:04:48.687904 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.687876 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d9ndd"] Jul 28 14:04:48.688070 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.688036 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" podUID="174e602e-e8de-414a-aa5d-ce6f5ac47700" containerName="redis" containerID="cri-o://9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79" gracePeriod=30 Jul 28 14:04:48.938720 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.938697 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:04:48.963864 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:48.963826 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" Jul 28 14:04:49.084524 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.084484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee123a15-39e9-4a1b-99fc-0d6940096f92-postgresdata\") pod \"ee123a15-39e9-4a1b-99fc-0d6940096f92\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " Jul 28 14:04:49.084724 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.084538 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhc29\" (UniqueName: \"kubernetes.io/projected/ee123a15-39e9-4a1b-99fc-0d6940096f92-kube-api-access-dhc29\") pod \"ee123a15-39e9-4a1b-99fc-0d6940096f92\" (UID: \"ee123a15-39e9-4a1b-99fc-0d6940096f92\") " Jul 28 14:04:49.084724 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.084621 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5nlz\" (UniqueName: \"kubernetes.io/projected/174e602e-e8de-414a-aa5d-ce6f5ac47700-kube-api-access-c5nlz\") pod \"174e602e-e8de-414a-aa5d-ce6f5ac47700\" (UID: \"174e602e-e8de-414a-aa5d-ce6f5ac47700\") " Jul 28 14:04:49.086849 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.086822 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee123a15-39e9-4a1b-99fc-0d6940096f92-kube-api-access-dhc29" (OuterVolumeSpecName: "kube-api-access-dhc29") pod "ee123a15-39e9-4a1b-99fc-0d6940096f92" (UID: "ee123a15-39e9-4a1b-99fc-0d6940096f92"). InnerVolumeSpecName "kube-api-access-dhc29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:04:49.086849 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.086845 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174e602e-e8de-414a-aa5d-ce6f5ac47700-kube-api-access-c5nlz" (OuterVolumeSpecName: "kube-api-access-c5nlz") pod "174e602e-e8de-414a-aa5d-ce6f5ac47700" (UID: "174e602e-e8de-414a-aa5d-ce6f5ac47700"). InnerVolumeSpecName "kube-api-access-c5nlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:04:49.117047 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.117004 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee123a15-39e9-4a1b-99fc-0d6940096f92-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "ee123a15-39e9-4a1b-99fc-0d6940096f92" (UID: "ee123a15-39e9-4a1b-99fc-0d6940096f92"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:04:49.147888 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.147847 2564 generic.go:356] "Generic (PLEG): container finished" podID="ee123a15-39e9-4a1b-99fc-0d6940096f92" containerID="aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd" exitCode=0 Jul 28 14:04:49.147888 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.147892 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" Jul 28 14:04:49.148467 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.147972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" event={"ID":"ee123a15-39e9-4a1b-99fc-0d6940096f92","Type":"ContainerDied","Data":"aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd"} Jul 28 14:04:49.148467 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.148012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-qgtlm" event={"ID":"ee123a15-39e9-4a1b-99fc-0d6940096f92","Type":"ContainerDied","Data":"34fc622c674807ec6ddbbf1eb055042c29363538b31b99cd3a2302ec29eed72f"} Jul 28 14:04:49.148467 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.148037 2564 scope.go:117] "RemoveContainer" containerID="aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd" Jul 28 14:04:49.149635 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.149605 2564 generic.go:356] "Generic (PLEG): container finished" podID="174e602e-e8de-414a-aa5d-ce6f5ac47700" containerID="9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79" exitCode=0 Jul 28 14:04:49.149759 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.149649 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" Jul 28 14:04:49.149759 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.149684 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" event={"ID":"174e602e-e8de-414a-aa5d-ce6f5ac47700","Type":"ContainerDied","Data":"9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79"} Jul 28 14:04:49.149759 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.149714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-d9ndd" event={"ID":"174e602e-e8de-414a-aa5d-ce6f5ac47700","Type":"ContainerDied","Data":"f07c99e7d1997005bf465ff905bee94283f5970982fd29fd1d2eec373d521291"} Jul 28 14:04:49.158414 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.158397 2564 scope.go:117] "RemoveContainer" containerID="aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd" Jul 28 14:04:49.160935 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:04:49.158696 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd\": container with ID starting with aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd not found: ID does not exist" containerID="aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd" Jul 28 14:04:49.160935 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.158735 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd"} err="failed to get container status \"aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd\": rpc error: code = NotFound desc = could not find container \"aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd\": container with ID starting with aede662b6e149227383f3f7fd00d592ed333e1946044ef4f8b4df927810576fd not found: ID does not exist" Jul 28 14:04:49.160935 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.158756 2564 scope.go:117] "RemoveContainer" containerID="9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79" Jul 28 14:04:49.170075 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.170033 2564 scope.go:117] "RemoveContainer" containerID="9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79" Jul 28 14:04:49.170424 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:04:49.170401 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79\": container with ID starting with 9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79 not found: ID does not exist" containerID="9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79" Jul 28 14:04:49.170503 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.170435 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79"} err="failed to get container status \"9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79\": rpc error: code = NotFound desc = could not find container \"9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79\": container with ID starting with 9c77ba2e8227cf6d18def6a2b67dbbbe9e232c8513b1699efac58e430ba6fd79 not found: ID does not exist" Jul 28 14:04:49.181354 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.181329 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d9ndd"] Jul 28 14:04:49.185277 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.185243 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5nlz\" (UniqueName: \"kubernetes.io/projected/174e602e-e8de-414a-aa5d-ce6f5ac47700-kube-api-access-c5nlz\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:04:49.185277 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.185277 2564 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ee123a15-39e9-4a1b-99fc-0d6940096f92-postgresdata\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:04:49.185421 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.185298 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc29\" (UniqueName: \"kubernetes.io/projected/ee123a15-39e9-4a1b-99fc-0d6940096f92-kube-api-access-dhc29\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:04:49.185498 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.185435 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-d9ndd"] Jul 28 14:04:49.201120 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.201085 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-qgtlm"] Jul 28 14:04:49.203186 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:49.203161 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-qgtlm"] Jul 28 14:04:50.781732 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:50.781700 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174e602e-e8de-414a-aa5d-ce6f5ac47700" path="/var/lib/kubelet/pods/174e602e-e8de-414a-aa5d-ce6f5ac47700/volumes" Jul 28 14:04:50.782105 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:50.782046 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee123a15-39e9-4a1b-99fc-0d6940096f92" path="/var/lib/kubelet/pods/ee123a15-39e9-4a1b-99fc-0d6940096f92/volumes" Jul 28 14:04:51.698568 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:51.698478 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 28 14:04:53.340699 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:04:53.340651 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 28 14:05:01.698680 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:01.698626 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 28 14:05:03.340964 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:03.340925 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 28 14:05:11.699033 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:11.698986 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" Jul 28 14:05:11.699418 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:11.699108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:05:13.341475 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:13.341433 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused" Jul 28 14:05:13.341943 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:13.341561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:05:17.260406 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.260371 2564 generic.go:356] "Generic (PLEG): container finished" podID="784a33fc-498b-4ece-8c59-6c299cf62651" containerID="10f5e0b09f1402a31af976dbb31c1eedb0ed95b729524d3bf7902956964d066e" exitCode=137 Jul 28 14:05:17.260818 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.260453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerDied","Data":"10f5e0b09f1402a31af976dbb31c1eedb0ed95b729524d3bf7902956964d066e"} Jul 28 14:05:17.262758 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.262735 2564 generic.go:356] "Generic (PLEG): container finished" podID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerID="977a58499e56ceb20c23115cc07e7da80e984e08b0797fe2a99cf18f4997ad79" exitCode=137 Jul 28 14:05:17.262870 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.262764 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerDied","Data":"977a58499e56ceb20c23115cc07e7da80e984e08b0797fe2a99cf18f4997ad79"} Jul 28 14:05:17.594486 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.594451 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:05:17.619276 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.619253 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:05:17.695047 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695017 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-registry-tls\") pod \"784a33fc-498b-4ece-8c59-6c299cf62651\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " Jul 28 14:05:17.695255 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695065 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26f6p\" (UniqueName: \"kubernetes.io/projected/b2c069cc-9528-4fb5-ad4e-c551d751af54-kube-api-access-26f6p\") pod \"b2c069cc-9528-4fb5-ad4e-c551d751af54\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " Jul 28 14:05:17.695255 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695102 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9mhp\" (UniqueName: \"kubernetes.io/projected/784a33fc-498b-4ece-8c59-6c299cf62651-kube-api-access-l9mhp\") pod \"784a33fc-498b-4ece-8c59-6c299cf62651\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " Jul 28 14:05:17.695255 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695117 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-registry-tls\") pod \"b2c069cc-9528-4fb5-ad4e-c551d751af54\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " Jul 28 14:05:17.695255 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695176 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls\") pod \"784a33fc-498b-4ece-8c59-6c299cf62651\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " Jul 28 14:05:17.695255 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695200 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c069cc-9528-4fb5-ad4e-c551d751af54-feast-data\") pod \"b2c069cc-9528-4fb5-ad4e-c551d751af54\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " Jul 28 14:05:17.695255 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695228 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/784a33fc-498b-4ece-8c59-6c299cf62651-feast-data\") pod \"784a33fc-498b-4ece-8c59-6c299cf62651\" (UID: \"784a33fc-498b-4ece-8c59-6c299cf62651\") " Jul 28 14:05:17.695255 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.695250 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-online-tls\") pod \"b2c069cc-9528-4fb5-ad4e-c551d751af54\" (UID: \"b2c069cc-9528-4fb5-ad4e-c551d751af54\") " Jul 28 14:05:17.696368 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.696331 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c069cc-9528-4fb5-ad4e-c551d751af54-feast-data" (OuterVolumeSpecName: "feast-data") pod "b2c069cc-9528-4fb5-ad4e-c551d751af54" (UID: "b2c069cc-9528-4fb5-ad4e-c551d751af54"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:05:17.697498 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.697472 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "784a33fc-498b-4ece-8c59-6c299cf62651" (UID: "784a33fc-498b-4ece-8c59-6c299cf62651"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:05:17.697880 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.697832 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b2c069cc-9528-4fb5-ad4e-c551d751af54" (UID: "b2c069cc-9528-4fb5-ad4e-c551d751af54"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:05:17.697880 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.697805 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-online-tls" (OuterVolumeSpecName: "online-tls") pod "b2c069cc-9528-4fb5-ad4e-c551d751af54" (UID: "b2c069cc-9528-4fb5-ad4e-c551d751af54"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:05:17.698022 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.697882 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c069cc-9528-4fb5-ad4e-c551d751af54-kube-api-access-26f6p" (OuterVolumeSpecName: "kube-api-access-26f6p") pod "b2c069cc-9528-4fb5-ad4e-c551d751af54" (UID: "b2c069cc-9528-4fb5-ad4e-c551d751af54"). InnerVolumeSpecName "kube-api-access-26f6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:05:17.698022 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.697934 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784a33fc-498b-4ece-8c59-6c299cf62651-kube-api-access-l9mhp" (OuterVolumeSpecName: "kube-api-access-l9mhp") pod "784a33fc-498b-4ece-8c59-6c299cf62651" (UID: "784a33fc-498b-4ece-8c59-6c299cf62651"). InnerVolumeSpecName "kube-api-access-l9mhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:05:17.698090 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.698041 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls" (OuterVolumeSpecName: "online-tls") pod "784a33fc-498b-4ece-8c59-6c299cf62651" (UID: "784a33fc-498b-4ece-8c59-6c299cf62651"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:05:17.704700 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.704675 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784a33fc-498b-4ece-8c59-6c299cf62651-feast-data" (OuterVolumeSpecName: "feast-data") pod "784a33fc-498b-4ece-8c59-6c299cf62651" (UID: "784a33fc-498b-4ece-8c59-6c299cf62651"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.796932 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-registry-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.796970 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26f6p\" (UniqueName: \"kubernetes.io/projected/b2c069cc-9528-4fb5-ad4e-c551d751af54-kube-api-access-26f6p\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.796989 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mhp\" (UniqueName: \"kubernetes.io/projected/784a33fc-498b-4ece-8c59-6c299cf62651-kube-api-access-l9mhp\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.797004 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-registry-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.797018 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/784a33fc-498b-4ece-8c59-6c299cf62651-online-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.797030 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c069cc-9528-4fb5-ad4e-c551d751af54-feast-data\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.797041 2564 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/784a33fc-498b-4ece-8c59-6c299cf62651-feast-data\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:17.797082 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:17.797051 2564 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2c069cc-9528-4fb5-ad4e-c551d751af54-online-tls\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:05:18.268620 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.268562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" event={"ID":"784a33fc-498b-4ece-8c59-6c299cf62651","Type":"ContainerDied","Data":"6634640a12449a9b59ee4934892ed8859a6324a6ce5fcbdc93c266e18e565d1b"} Jul 28 14:05:18.269136 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.268653 2564 scope.go:117] "RemoveContainer" containerID="4a6d050ba8bb445bd7bf329d44007a49b7ec40c8cc674051821d65d32125a0a2" Jul 28 14:05:18.269136 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.268600 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg" Jul 28 14:05:18.270926 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.270905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" event={"ID":"b2c069cc-9528-4fb5-ad4e-c551d751af54","Type":"ContainerDied","Data":"25d6940ed48bb9ec6125a997b06126999afe7178111a0a8bb0e442d8e80e7954"} Jul 28 14:05:18.271028 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.270947 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c" Jul 28 14:05:18.278191 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.278084 2564 scope.go:117] "RemoveContainer" containerID="10f5e0b09f1402a31af976dbb31c1eedb0ed95b729524d3bf7902956964d066e" Jul 28 14:05:18.288955 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.288934 2564 scope.go:117] "RemoveContainer" containerID="d7faab0556e00e2d3fb47acc6cf7fc5cae5d79f1739d9642f8b8accf313af257" Jul 28 14:05:18.295672 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.295645 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg"] Jul 28 14:05:18.296871 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.296846 2564 scope.go:117] "RemoveContainer" containerID="b6085c0b857c471c82754cf6996d01dad9ade29e9f449f75acb60b86f17d12e0" Jul 28 14:05:18.300282 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.300261 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5c6f98c969-xpfkg"] Jul 28 14:05:18.304489 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.304474 2564 scope.go:117] "RemoveContainer" containerID="833dbdcf91f678e4167c9236232fec05508af3309460d0c36ab28dc98a68fba1" Jul 28 14:05:18.311891 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.311869 2564 scope.go:117] "RemoveContainer" containerID="977a58499e56ceb20c23115cc07e7da80e984e08b0797fe2a99cf18f4997ad79" Jul 28 14:05:18.317226 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.317197 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c"] Jul 28 14:05:18.319515 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.319499 2564 scope.go:117] "RemoveContainer" containerID="5b826fd2ee625c388b7b3854702d5cfb6335454f9b81b4ece47ddcbeb0367f46" Jul 28 14:05:18.323792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.323772 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6479d5dc78-kxj2c"] Jul 28 14:05:18.327411 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.327394 2564 scope.go:117] "RemoveContainer" containerID="0620ffda6181f1d97a69b4035db86eaaa73b6a7900bdf97b2424306b392231fe" Jul 28 14:05:18.782547 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.782516 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" path="/var/lib/kubelet/pods/784a33fc-498b-4ece-8c59-6c299cf62651/volumes" Jul 28 14:05:18.783050 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:05:18.783034 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" path="/var/lib/kubelet/pods/b2c069cc-9528-4fb5-ad4e-c551d751af54/volumes" Jul 28 14:16:05.347282 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347246 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crg7m/must-gather-kv78s"] Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347533 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174e602e-e8de-414a-aa5d-ce6f5ac47700" containerName="redis" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347543 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="174e602e-e8de-414a-aa5d-ce6f5ac47700" containerName="redis" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347551 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="feast-init" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347557 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="feast-init" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347564 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee123a15-39e9-4a1b-99fc-0d6940096f92" containerName="postgres" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347570 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee123a15-39e9-4a1b-99fc-0d6940096f92" containerName="postgres" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347597 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347606 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347619 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="registry" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347627 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="registry" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347638 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="feast-apply" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347645 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="feast-apply" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347655 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="feast-apply" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347662 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="feast-apply" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347674 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="registry" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347679 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="registry" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347684 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347689 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347696 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="feast-init" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347701 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="feast-init" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347749 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="174e602e-e8de-414a-aa5d-ce6f5ac47700" containerName="redis" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347756 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="registry" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347763 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee123a15-39e9-4a1b-99fc-0d6940096f92" containerName="postgres" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347769 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="registry" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347774 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c069cc-9528-4fb5-ad4e-c551d751af54" containerName="online" Jul 28 14:16:05.347792 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.347780 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="784a33fc-498b-4ece-8c59-6c299cf62651" containerName="online" Jul 28 14:16:05.350613 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.350597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.353315 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.353292 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crg7m\"/\"openshift-service-ca.crt\"" Jul 28 14:16:05.353537 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.353520 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crg7m\"/\"kube-root-ca.crt\"" Jul 28 14:16:05.354363 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.354349 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-crg7m\"/\"default-dockercfg-zclgt\"" Jul 28 14:16:05.359500 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.359471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crg7m/must-gather-kv78s"] Jul 28 14:16:05.412314 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.412274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ac7b215-2c83-462a-8075-4ce99fd28d91-must-gather-output\") pod \"must-gather-kv78s\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.412484 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.412337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnkc\" (UniqueName: \"kubernetes.io/projected/3ac7b215-2c83-462a-8075-4ce99fd28d91-kube-api-access-4xnkc\") pod \"must-gather-kv78s\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.513602 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.513539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnkc\" (UniqueName: \"kubernetes.io/projected/3ac7b215-2c83-462a-8075-4ce99fd28d91-kube-api-access-4xnkc\") pod \"must-gather-kv78s\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.513770 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.513640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ac7b215-2c83-462a-8075-4ce99fd28d91-must-gather-output\") pod \"must-gather-kv78s\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.513993 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.513977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ac7b215-2c83-462a-8075-4ce99fd28d91-must-gather-output\") pod \"must-gather-kv78s\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.524441 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.524415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnkc\" (UniqueName: \"kubernetes.io/projected/3ac7b215-2c83-462a-8075-4ce99fd28d91-kube-api-access-4xnkc\") pod \"must-gather-kv78s\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.660534 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.660443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:05.787325 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.787288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crg7m/must-gather-kv78s"] Jul 28 14:16:05.790020 ip-10-0-131-116 kubenswrapper[2564]: W0728 14:16:05.789992 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ac7b215_2c83_462a_8075_4ce99fd28d91.slice/crio-62d835b409c509a39758b824e39c05533a505ae5df4134d91f54ce35aa53c700 WatchSource:0}: Error finding container 62d835b409c509a39758b824e39c05533a505ae5df4134d91f54ce35aa53c700: Status 404 returned error can't find the container with id 62d835b409c509a39758b824e39c05533a505ae5df4134d91f54ce35aa53c700 Jul 28 14:16:05.791744 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:05.791725 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 14:16:06.602467 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:06.602421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crg7m/must-gather-kv78s" event={"ID":"3ac7b215-2c83-462a-8075-4ce99fd28d91","Type":"ContainerStarted","Data":"62d835b409c509a39758b824e39c05533a505ae5df4134d91f54ce35aa53c700"} Jul 28 14:16:12.631020 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:12.630991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crg7m/must-gather-kv78s" event={"ID":"3ac7b215-2c83-462a-8075-4ce99fd28d91","Type":"ContainerStarted","Data":"dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0"} Jul 28 14:16:12.631388 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:12.631033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crg7m/must-gather-kv78s" event={"ID":"3ac7b215-2c83-462a-8075-4ce99fd28d91","Type":"ContainerStarted","Data":"4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0"} Jul 28 14:16:12.653398 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:12.653351 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crg7m/must-gather-kv78s" podStartSLOduration=1.212944642 podStartE2EDuration="7.653336762s" totalImagesPullingTime="6.44039212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:16:12.65123674 +0000 UTC m=+2024.352780068" watchObservedRunningTime="2026-07-28 14:16:12.653336762 +0000 UTC m=+2024.354880089" Jul 28 14:16:22.673151 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:22.673111 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerID="4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0" exitCode=0 Jul 28 14:16:22.673575 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:22.673187 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crg7m/must-gather-kv78s" event={"ID":"3ac7b215-2c83-462a-8075-4ce99fd28d91","Type":"ContainerDied","Data":"4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0"} Jul 28 14:16:22.673575 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:22.673478 2564 scope.go:117] "RemoveContainer" containerID="4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0" Jul 28 14:16:22.913154 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:22.913124 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-crg7m_must-gather-kv78s_3ac7b215-2c83-462a-8075-4ce99fd28d91/gather/0.log" Jul 28 14:16:26.553739 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:26.553707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cmjgg_9d7a2dec-a85e-4a73-b3ad-245c59d03c26/global-pull-secret-syncer/0.log" Jul 28 14:16:26.656862 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:26.656833 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h4qjv_b689bc81-3feb-45f2-b076-aa4278074c6e/konnectivity-agent/0.log" Jul 28 14:16:26.807281 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:26.807194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-116.ec2.internal_675d2604a48b7f6d7a7c5b5b942d4938/haproxy/0.log" Jul 28 14:16:28.271331 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.271297 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-crg7m/must-gather-kv78s"] Jul 28 14:16:28.271800 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.271474 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-crg7m/must-gather-kv78s" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerName="copy" containerID="cri-o://dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0" gracePeriod=2 Jul 28 14:16:28.286501 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.286475 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-crg7m/must-gather-kv78s"] Jul 28 14:16:28.288978 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:16:28.288941 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3ac7b215-2c83-462a-8075-4ce99fd28d91\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-crg7m\"/\"must-gather-kv78s\": pods \"must-gather-kv78s\" not found" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:28.499894 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.499869 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-crg7m_must-gather-kv78s_3ac7b215-2c83-462a-8075-4ce99fd28d91/copy/0.log" Jul 28 14:16:28.502389 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.502372 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:28.504959 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:16:28.504934 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kv78s\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-crg7m\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:28.613873 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.613784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ac7b215-2c83-462a-8075-4ce99fd28d91-must-gather-output\") pod \"3ac7b215-2c83-462a-8075-4ce99fd28d91\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " Jul 28 14:16:28.613873 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.613852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xnkc\" (UniqueName: \"kubernetes.io/projected/3ac7b215-2c83-462a-8075-4ce99fd28d91-kube-api-access-4xnkc\") pod \"3ac7b215-2c83-462a-8075-4ce99fd28d91\" (UID: \"3ac7b215-2c83-462a-8075-4ce99fd28d91\") " Jul 28 14:16:28.614271 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.614236 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac7b215-2c83-462a-8075-4ce99fd28d91-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "3ac7b215-2c83-462a-8075-4ce99fd28d91" (UID: "3ac7b215-2c83-462a-8075-4ce99fd28d91"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:16:28.616069 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.616040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac7b215-2c83-462a-8075-4ce99fd28d91-kube-api-access-4xnkc" (OuterVolumeSpecName: "kube-api-access-4xnkc") pod "3ac7b215-2c83-462a-8075-4ce99fd28d91" (UID: "3ac7b215-2c83-462a-8075-4ce99fd28d91"). InnerVolumeSpecName "kube-api-access-4xnkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:16:28.697840 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.697812 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-crg7m_must-gather-kv78s_3ac7b215-2c83-462a-8075-4ce99fd28d91/copy/0.log" Jul 28 14:16:28.698155 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.698135 2564 generic.go:356] "Generic (PLEG): container finished" podID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerID="dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0" exitCode=143 Jul 28 14:16:28.698217 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.698189 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:28.698254 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.698227 2564 scope.go:117] "RemoveContainer" containerID="dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0" Jul 28 14:16:28.701271 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:16:28.701246 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kv78s\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-crg7m\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:28.705950 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.705664 2564 scope.go:117] "RemoveContainer" containerID="4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0" Jul 28 14:16:28.708639 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:16:28.708615 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kv78s\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-crg7m\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:28.714963 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.714933 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xnkc\" (UniqueName: \"kubernetes.io/projected/3ac7b215-2c83-462a-8075-4ce99fd28d91-kube-api-access-4xnkc\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:16:28.714963 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.714958 2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ac7b215-2c83-462a-8075-4ce99fd28d91-must-gather-output\") on node \"ip-10-0-131-116.ec2.internal\" DevicePath \"\"" Jul 28 14:16:28.717621 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.717603 2564 scope.go:117] "RemoveContainer" containerID="dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0" Jul 28 14:16:28.717925 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:16:28.717903 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0\": container with ID starting with dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0 not found: ID does not exist" containerID="dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0" Jul 28 14:16:28.717965 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.717938 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0"} err="failed to get container status \"dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0\": rpc error: code = NotFound desc = could not find container \"dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0\": container with ID starting with dfe38453abf3bd408546e2dad80a07432da873c406057b8747bca1bb79e495e0 not found: ID does not exist" Jul 28 14:16:28.717965 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.717957 2564 scope.go:117] "RemoveContainer" containerID="4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0" Jul 28 14:16:28.718211 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:16:28.718194 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0\": container with ID starting with 4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0 not found: ID does not exist" containerID="4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0" Jul 28 14:16:28.718249 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.718221 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0"} err="failed to get container status \"4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0\": rpc error: code = NotFound desc = could not find container \"4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0\": container with ID starting with 4f0670e8cfeaea084706a793c19b92ca1b2376e92de75ae7d6aa076e252eb9d0 not found: ID does not exist" Jul 28 14:16:28.781197 ip-10-0-131-116 kubenswrapper[2564]: E0728 14:16:28.781158 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-kv78s\" is forbidden: User \"system:node:ip-10-0-131-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-crg7m\": no relationship found between node 'ip-10-0-131-116.ec2.internal' and this object" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" pod="openshift-must-gather-crg7m/must-gather-kv78s" Jul 28 14:16:28.782270 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:28.782246 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" path="/var/lib/kubelet/pods/3ac7b215-2c83-462a-8075-4ce99fd28d91/volumes" Jul 28 14:16:30.632800 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:30.632767 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-j6d4s_dcc88e5c-0cde-4f78-9d31-34c61b544946/cluster-monitoring-operator/0.log" Jul 28 14:16:30.688793 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:30.688762 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xvzcw_735b6e9e-3349-4c06-8277-eb78764914b9/kube-state-metrics/0.log" Jul 28 14:16:30.717623 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:30.717572 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xvzcw_735b6e9e-3349-4c06-8277-eb78764914b9/kube-rbac-proxy-main/0.log" Jul 28 14:16:30.743535 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:30.743503 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xvzcw_735b6e9e-3349-4c06-8277-eb78764914b9/kube-rbac-proxy-self/0.log" Jul 28 14:16:30.826595 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:30.826544 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-2g5vg_420193a6-0784-4014-aa0a-8a35eba222bc/monitoring-plugin/0.log" Jul 28 14:16:31.201185 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:31.201155 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w58qm_08f147d7-5099-4a2d-a918-d92893c355ff/node-exporter/0.log" Jul 28 14:16:31.245047 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:31.245018 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w58qm_08f147d7-5099-4a2d-a918-d92893c355ff/kube-rbac-proxy/0.log" Jul 28 14:16:31.304831 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:31.304804 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w58qm_08f147d7-5099-4a2d-a918-d92893c355ff/init-textfile/0.log" Jul 28 14:16:31.368980 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:31.368947 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rhprx_514a877e-f2e4-4330-9206-eb8dd8fd20d9/kube-rbac-proxy-main/0.log" Jul 28 14:16:31.422040 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:31.422013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rhprx_514a877e-f2e4-4330-9206-eb8dd8fd20d9/kube-rbac-proxy-self/0.log" Jul 28 14:16:31.471323 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:31.471241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rhprx_514a877e-f2e4-4330-9206-eb8dd8fd20d9/openshift-state-metrics/0.log" Jul 28 14:16:32.167332 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:32.167297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cb6f8c64-8nrmt_6ef0b607-e986-42d7-8e8c-6c5fd0557b93/telemeter-client/0.log" Jul 28 14:16:32.207377 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:32.207351 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cb6f8c64-8nrmt_6ef0b607-e986-42d7-8e8c-6c5fd0557b93/reload/0.log" Jul 28 14:16:32.249050 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:32.249021 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56cb6f8c64-8nrmt_6ef0b607-e986-42d7-8e8c-6c5fd0557b93/kube-rbac-proxy/0.log" Jul 28 14:16:33.433167 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.433130 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr"] Jul 28 14:16:33.433554 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.433416 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerName="gather" Jul 28 14:16:33.433554 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.433427 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerName="gather" Jul 28 14:16:33.433554 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.433436 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerName="copy" Jul 28 14:16:33.433554 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.433441 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerName="copy" Jul 28 14:16:33.433554 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.433489 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerName="gather" Jul 28 14:16:33.433554 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.433497 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac7b215-2c83-462a-8075-4ce99fd28d91" containerName="copy" Jul 28 14:16:33.436590 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.436557 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.440157 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.440134 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hn42g\"/\"kube-root-ca.crt\"" Jul 28 14:16:33.441365 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.441347 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hn42g\"/\"default-dockercfg-dfdvd\"" Jul 28 14:16:33.441365 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.441359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hn42g\"/\"openshift-service-ca.crt\"" Jul 28 14:16:33.448487 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.448459 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr"] Jul 28 14:16:33.526870 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.526836 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-sg7c6_24274456-35d5-411f-a0a3-bad2157ea5a6/networking-console-plugin/0.log" Jul 28 14:16:33.555410 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.555379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-podres\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.555572 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.555421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-lib-modules\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.555572 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.555453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-sys\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.555572 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.555509 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-proc\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.555572 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.555550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nbw\" (UniqueName: \"kubernetes.io/projected/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-kube-api-access-w7nbw\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.655974 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.655932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-proc\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.655974 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.655982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nbw\" (UniqueName: \"kubernetes.io/projected/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-kube-api-access-w7nbw\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.656171 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.656041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-podres\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.656171 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.656063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-lib-modules\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.656171 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.656079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-proc\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.656171 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.656103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-sys\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.656326 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.656174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-sys\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.656326 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.656190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-podres\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.656326 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.656219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-lib-modules\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.665746 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.665710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nbw\" (UniqueName: \"kubernetes.io/projected/5ee9aa30-ba15-4b92-8fa6-efb6b312b86d-kube-api-access-w7nbw\") pod \"perf-node-gather-daemonset-7cnqr\" (UID: \"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d\") " pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.746575 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.746541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:33.880013 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:33.879940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr"] Jul 28 14:16:33.882548 ip-10-0-131-116 kubenswrapper[2564]: W0728 14:16:33.882516 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ee9aa30_ba15_4b92_8fa6_efb6b312b86d.slice/crio-34a6533b95fe23d72a6b5a243409699ce7285b08a64a62eca79b2744876c0859 WatchSource:0}: Error finding container 34a6533b95fe23d72a6b5a243409699ce7285b08a64a62eca79b2744876c0859: Status 404 returned error can't find the container with id 34a6533b95fe23d72a6b5a243409699ce7285b08a64a62eca79b2744876c0859 Jul 28 14:16:34.394012 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:34.393977 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-krsgb_72bf710f-32e1-40ac-94dc-361188ca94aa/download-server/0.log" Jul 28 14:16:34.722364 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:34.722262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" event={"ID":"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d","Type":"ContainerStarted","Data":"2744e23d1b002ae9a5c249b9b13a489e282d9405784b30551cecada2fdcd1110"} Jul 28 14:16:34.722364 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:34.722354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" event={"ID":"5ee9aa30-ba15-4b92-8fa6-efb6b312b86d","Type":"ContainerStarted","Data":"34a6533b95fe23d72a6b5a243409699ce7285b08a64a62eca79b2744876c0859"} Jul 28 14:16:34.722808 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:34.722402 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:34.741292 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:34.741240 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" podStartSLOduration=1.741224248 podStartE2EDuration="1.741224248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:16:34.740082604 +0000 UTC m=+2046.441625933" watchObservedRunningTime="2026-07-28 14:16:34.741224248 +0000 UTC m=+2046.442767576" Jul 28 14:16:35.772368 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:35.772335 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsw4v_2bc33d9d-f9d6-40b3-83ce-dc1aebf54246/dns/0.log" Jul 28 14:16:35.797251 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:35.797220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsw4v_2bc33d9d-f9d6-40b3-83ce-dc1aebf54246/kube-rbac-proxy/0.log" Jul 28 14:16:35.825388 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:35.825358 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wvn7w_550041e0-5515-4eeb-98b2-23273b7c848d/dns-node-resolver/0.log" Jul 28 14:16:36.315132 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:36.315099 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-859d744f44-476ss_f8fe3421-0e3b-4a57-8648-8bfe1a0d7d25/registry/0.log" Jul 28 14:16:36.408181 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:36.408137 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tz84d_13431ed3-6ada-4d5d-b763-b30d0fd9c039/node-ca/0.log" Jul 28 14:16:37.573691 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:37.573652 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2b7d_99e40087-a96f-4656-8059-eae9600913e5/serve-healthcheck-canary/0.log" Jul 28 14:16:38.059631 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:38.059597 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lxwmm_660b57a2-689c-49b9-a48d-5bebc4138ff5/kube-rbac-proxy/0.log" Jul 28 14:16:38.080838 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:38.080803 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lxwmm_660b57a2-689c-49b9-a48d-5bebc4138ff5/exporter/0.log" Jul 28 14:16:38.103911 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:38.103886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lxwmm_660b57a2-689c-49b9-a48d-5bebc4138ff5/extractor/0.log" Jul 28 14:16:40.735947 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:40.735922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hn42g/perf-node-gather-daemonset-7cnqr" Jul 28 14:16:44.227216 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.227188 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rqqk5_92cf0d6f-c2e8-4494-a431-1a339f34f906/kube-multus-additional-cni-plugins/0.log" Jul 28 14:16:44.256303 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.256220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rqqk5_92cf0d6f-c2e8-4494-a431-1a339f34f906/egress-router-binary-copy/0.log" Jul 28 14:16:44.281852 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.281805 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rqqk5_92cf0d6f-c2e8-4494-a431-1a339f34f906/cni-plugins/0.log" Jul 28 14:16:44.307062 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.307033 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rqqk5_92cf0d6f-c2e8-4494-a431-1a339f34f906/bond-cni-plugin/0.log" Jul 28 14:16:44.334345 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.334316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rqqk5_92cf0d6f-c2e8-4494-a431-1a339f34f906/routeoverride-cni/0.log" Jul 28 14:16:44.358995 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.358967 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rqqk5_92cf0d6f-c2e8-4494-a431-1a339f34f906/whereabouts-cni-bincopy/0.log" Jul 28 14:16:44.382980 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.382947 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rqqk5_92cf0d6f-c2e8-4494-a431-1a339f34f906/whereabouts-cni/0.log" Jul 28 14:16:44.648049 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.648015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k7qpg_135af29a-f0e3-4c23-beb7-f428f6883fa9/kube-multus/0.log" Jul 28 14:16:44.709038 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.709001 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k4k9j_6f6e6450-cd8c-4979-90e5-06011a1d268c/network-metrics-daemon/0.log" Jul 28 14:16:44.731353 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:44.731323 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k4k9j_6f6e6450-cd8c-4979-90e5-06011a1d268c/kube-rbac-proxy/0.log" Jul 28 14:16:45.555285 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.555251 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/ovn-controller/0.log" Jul 28 14:16:45.575488 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.575460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/ovn-acl-logging/0.log" Jul 28 14:16:45.605017 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.604988 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/kube-rbac-proxy-node/0.log" Jul 28 14:16:45.631002 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.630969 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 14:16:45.661080 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.661046 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/northd/0.log" Jul 28 14:16:45.689420 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.689391 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/nbdb/0.log" Jul 28 14:16:45.723073 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.723044 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/sbdb/0.log" Jul 28 14:16:45.787407 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:45.787373 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5cdj_e2964eed-97bc-4f82-a21e-66909e93e934/ovnkube-controller/0.log" Jul 28 14:16:47.438532 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:47.438497 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-r6j7c_24616ef5-ef24-4816-a359-07511a2e6c75/network-check-target-container/0.log" Jul 28 14:16:48.340932 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:48.340897 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6ftcf_ab63b546-6312-4b33-be18-8d788451865b/iptables-alerter/0.log" Jul 28 14:16:49.121080 ip-10-0-131-116 kubenswrapper[2564]: I0728 14:16:49.121045 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-h2qwd_cfccaf3f-da3c-430d-9640-c91af8e694f3/tuned/0.log"