Jul 29 08:19:18.511700 ip-10-0-138-229 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 08:19:18.511711 ip-10-0-138-229 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 08:19:18.511718 ip-10-0-138-229 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 08:19:18.511987 ip-10-0-138-229 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 08:19:28.746343 ip-10-0-138-229 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 08:19:28.746360 ip-10-0-138-229 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 86bef5fa953742e2bd59fd54d43ba940 -- Jul 29 08:21:03.935393 ip-10-0-138-229 systemd[1]: Starting Kubernetes Kubelet... Jul 29 08:21:04.289185 ip-10-0-138-229 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:21:04.289185 ip-10-0-138-229 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 08:21:04.289185 ip-10-0-138-229 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:21:04.289185 ip-10-0-138-229 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 08:21:04.289185 ip-10-0-138-229 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:21:04.290236 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.290159 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 08:21:04.294097 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294083 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:21:04.294097 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294098 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294101 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294105 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294108 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294111 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294114 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294117 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294120 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294123 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294125 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294128 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294137 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294140 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294143 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294146 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294148 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294151 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294153 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294156 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294158 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:21:04.294162 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294161 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294164 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294166 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294169 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294172 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294175 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294178 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294180 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294182 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294185 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294187 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294189 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294192 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294194 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294196 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294199 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294201 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294211 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294213 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294216 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:21:04.294637 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294218 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294220 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294223 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294226 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294228 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294230 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294233 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294235 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294237 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294240 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294242 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294245 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294247 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294249 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294252 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294256 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294258 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294261 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294264 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:21:04.295132 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294268 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294271 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294275 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294278 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294281 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294284 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294287 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294289 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294293 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294297 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294300 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294303 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294306 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294309 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294312 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294314 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294317 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294319 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294321 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:21:04.295607 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294325 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294328 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294330 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294333 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294335 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294337 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294340 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294343 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294346 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294348 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294351 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294354 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294356 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294359 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294361 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294365 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294368 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294371 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294373 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:21:04.296099 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.294376 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.295995 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296003 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296007 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296010 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296013 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296017 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296020 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296025 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296028 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296031 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296035 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296039 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296042 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296045 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296048 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296051 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296053 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296056 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:21:04.296559 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296058 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296061 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296064 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296067 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296070 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296073 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296076 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296078 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296081 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296084 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296086 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296089 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296091 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296093 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296096 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296098 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296101 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296104 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296106 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:21:04.297120 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296109 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296113 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296118 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296121 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296124 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296127 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296130 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296132 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296135 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296137 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296140 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296142 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296144 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296147 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296150 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296152 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296155 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296158 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296161 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:21:04.297612 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296163 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296166 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296168 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296170 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296173 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296175 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296178 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296181 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296185 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296189 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296191 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296194 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296196 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296199 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296202 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296206 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296208 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296211 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296213 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296216 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:21:04.298094 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296219 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296221 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296224 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296226 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296228 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296231 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296233 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296236 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296238 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296241 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296244 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296246 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296249 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296251 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296253 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296256 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296258 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296261 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296263 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296265 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:21:04.298576 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296268 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296270 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296272 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296620 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296624 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296627 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296630 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296634 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296637 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296639 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296642 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296644 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296647 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296649 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296652 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296654 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296657 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296659 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296662 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296664 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:21:04.299079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296667 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296670 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296672 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296675 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296678 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296680 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296683 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296685 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296687 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296690 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296692 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296695 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296697 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296700 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296702 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296705 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296708 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296710 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296713 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296716 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:21:04.299552 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296719 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296722 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296724 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296727 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296729 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296732 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296734 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296737 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296739 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296742 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296744 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296747 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296751 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296754 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296757 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296760 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296763 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296765 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296770 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:21:04.300042 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296773 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296775 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296777 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296780 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296782 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296785 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296787 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296790 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296792 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296795 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296797 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296800 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296802 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296805 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296807 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296810 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296813 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296815 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296818 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296820 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:21:04.300506 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296823 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296825 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296829 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296832 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296835 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296837 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296840 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296842 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296845 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296864 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296866 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296871 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296876 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296879 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296881 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296884 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296886 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296889 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296891 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:21:04.301007 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296894 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296896 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296899 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.296901 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.296971 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.296990 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.296996 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297000 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297005 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297009 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297013 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297017 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297020 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297023 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297026 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297029 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297032 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297034 2580 flags.go:64] FLAG: --cgroup-root="" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297037 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297040 2580 flags.go:64] FLAG: --client-ca-file="" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297043 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297046 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297050 2580 flags.go:64] FLAG: --cluster-domain="" Jul 29 08:21:04.301481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297052 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297057 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297061 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297064 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297068 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297071 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297074 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297077 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297080 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297082 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297085 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297088 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297091 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297095 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297098 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297101 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297103 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297107 2580 flags.go:64] FLAG: --enable-server="true" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297110 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297114 2580 flags.go:64] FLAG: --event-burst="100" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297117 2580 flags.go:64] FLAG: --event-qps="50" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297121 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297124 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297127 2580 flags.go:64] FLAG: --eviction-hard="" Jul 29 08:21:04.302092 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297130 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297133 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297136 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297139 2580 flags.go:64] FLAG: --eviction-soft="" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297142 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297145 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297148 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297151 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297153 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297157 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297159 2580 flags.go:64] FLAG: --feature-gates="" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297163 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297166 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297169 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297172 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297175 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297178 2580 flags.go:64] FLAG: --help="false" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297180 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297183 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297187 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297189 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297193 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297196 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297198 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 08:21:04.302919 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297201 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297204 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297207 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297210 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297213 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297216 2580 flags.go:64] FLAG: --kube-reserved="" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297218 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297221 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297224 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297226 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297229 2580 flags.go:64] FLAG: --lock-file="" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297232 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297238 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297241 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297246 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297249 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297251 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297254 2580 flags.go:64] FLAG: --logging-format="text" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297257 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297260 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297263 2580 flags.go:64] FLAG: --manifest-url="" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297266 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297270 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297273 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297277 2580 flags.go:64] FLAG: --max-pods="110" Jul 29 08:21:04.303527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297279 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297282 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297285 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297288 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297291 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297294 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297297 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297303 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297306 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297309 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297312 2580 flags.go:64] FLAG: --pod-cidr="" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297315 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297320 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297323 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297326 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297328 2580 flags.go:64] FLAG: --port="10250" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297332 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297334 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b903183bfeb5ff72" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297337 2580 flags.go:64] FLAG: --qos-reserved="" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297340 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297344 2580 flags.go:64] FLAG: --register-node="true" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297347 2580 flags.go:64] FLAG: --register-with-taints="" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297350 2580 flags.go:64] FLAG: --registry-burst="10" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297353 2580 flags.go:64] FLAG: --registry-qps="5" Jul 29 08:21:04.304165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297356 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297359 2580 flags.go:64] FLAG: --reserved-memory="" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297362 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297365 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297368 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297371 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297374 2580 flags.go:64] FLAG: --runonce="false" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297376 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297379 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297382 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297384 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297387 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297390 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297393 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297396 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297398 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297401 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297404 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297407 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297409 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297413 2580 flags.go:64] FLAG: --system-cgroups="" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297415 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297420 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297424 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297426 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 08:21:04.304750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297431 2580 flags.go:64] FLAG: --tls-min-version="" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297433 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297436 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297440 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297443 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297446 2580 flags.go:64] FLAG: --v="2" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297452 2580 flags.go:64] FLAG: --version="false" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297456 2580 flags.go:64] FLAG: --vmodule="" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297459 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.297463 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297614 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297628 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297631 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297634 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297637 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297641 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297645 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297647 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297651 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297653 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297656 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297659 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:21:04.305408 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297662 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297665 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297667 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297670 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297672 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297675 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297678 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297680 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297683 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297685 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297688 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297690 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297693 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297695 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297698 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297700 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297703 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297705 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297708 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297710 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:21:04.305966 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297713 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297719 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297722 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297724 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297728 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297733 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297736 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297739 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297741 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297744 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297747 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297749 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297752 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297754 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297757 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297759 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297762 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297765 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297767 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297770 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:21:04.306539 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297773 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297775 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297778 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297780 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297782 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297785 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297788 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297790 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297793 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297796 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297798 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297801 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297804 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297809 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297812 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297814 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297817 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297819 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297822 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297824 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:21:04.307059 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297827 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297830 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297832 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297834 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297837 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297839 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297842 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297844 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297847 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297862 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297865 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297867 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297871 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297875 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297878 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297881 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297884 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297886 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297889 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:21:04.307537 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297891 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297894 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297896 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297899 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297901 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297905 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297911 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.297914 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.298724 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.306050 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.306068 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.306989 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.306997 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307048 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:21:04.308066 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307054 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307057 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307060 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307063 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307066 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307069 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307073 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307076 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307078 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307081 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307083 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307086 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307088 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307091 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307095 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307099 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307102 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307104 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307107 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307109 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:21:04.308411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307112 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307115 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307118 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307120 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307123 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307126 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307128 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307131 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307135 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307138 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307140 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307143 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307145 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307148 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307150 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307153 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307155 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307158 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307160 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307162 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307165 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:21:04.308922 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307167 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307170 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307172 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307174 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307177 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307180 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307182 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307184 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307187 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307189 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307192 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307194 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307197 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307199 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307202 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307204 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307208 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307211 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307214 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307217 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:21:04.309635 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307220 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307223 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307226 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307228 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307230 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307233 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307235 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307238 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307240 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307243 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307246 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307249 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307252 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307254 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307257 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307259 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307262 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307264 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307267 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:21:04.310390 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307269 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307271 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307274 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307276 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307279 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307281 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307283 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307286 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307288 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307290 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307293 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307296 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307299 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307301 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307305 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307307 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307309 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:21:04.310971 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307312 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.307317 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307435 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307439 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307442 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307445 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307448 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307450 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307453 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307456 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307459 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307461 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307464 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307466 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307469 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:21:04.311377 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307471 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307473 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307476 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307478 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307481 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307483 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307485 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307488 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307490 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307493 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307495 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307499 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307502 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307505 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307508 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307511 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307514 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307517 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307519 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:21:04.311744 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307522 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307524 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307527 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307529 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307532 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307534 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307537 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307540 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307543 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307546 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307548 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307551 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307553 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307556 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307558 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307560 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307563 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307566 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307569 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307571 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:21:04.312348 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307574 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307576 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307579 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307581 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307583 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307586 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307589 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307591 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307594 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307597 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307599 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307602 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307604 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307607 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307609 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307611 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307614 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307616 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307619 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307621 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:21:04.312897 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307624 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307627 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307629 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307631 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307634 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307637 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307640 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307643 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307645 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307647 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307650 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307652 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307655 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307657 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307660 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307662 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307664 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307667 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307669 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307672 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:21:04.313411 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307674 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307676 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307679 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307684 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307686 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307688 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:04.307692 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.307697 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.307816 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 08:21:04.313908 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.312384 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 08:21:04.314144 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.314103 2580 server.go:1019] "Starting client certificate rotation" Jul 29 08:21:04.314225 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.314206 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 08:21:04.314277 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.314247 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 08:21:04.316988 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.316966 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 08:21:04.318113 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.318092 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 08:21:04.329913 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.329897 2580 log.go:25] "Validated CRI v1 runtime API" Jul 29 08:21:04.332227 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.332213 2580 log.go:25] "Validated CRI v1 image API" Jul 29 08:21:04.332998 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.332984 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 08:21:04.335601 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.335585 2580 fs.go:135] Filesystem UUIDs: map[264c27be-38e3-4062-8a78-3c64ff090c7e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b76a9442-cd91-480f-bbc6-885ebdfbea2a:/dev/nvme0n1p4] Jul 29 08:21:04.335646 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.335602 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 08:21:04.340654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.340553 2580 manager.go:216] Machine: {Timestamp:2026-07-29 08:21:04.34010004 +0000 UTC m=+0.292362850 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098070 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f4af94bc5443db3c5bff3dadd3f0b SystemUUID:ec2f4af9-4bc5-443d-b3c5-bff3dadd3f0b BootID:86bef5fa-9537-42e2-bd59-fd54d43ba940 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:b9:47:d8:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:b9:47:d8:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:4d:c1:49:9c:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 08:21:04.340654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.340652 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 08:21:04.340766 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.340718 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 08:21:04.344255 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.344233 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 08:21:04.344406 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.344256 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 08:21:04.344482 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.344468 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 08:21:04.345160 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.345149 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 08:21:04.345203 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.345163 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 08:21:04.345672 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.345662 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 08:21:04.346532 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.346521 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 08:21:04.348553 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.348543 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 08:21:04.352256 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352245 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 29 08:21:04.352371 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352362 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 08:21:04.352403 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352390 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 29 08:21:04.352432 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352406 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 08:21:04.352432 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352426 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 08:21:04.352482 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352434 2580 kubelet.go:398] "Adding apiserver pod source" Jul 29 08:21:04.352482 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352442 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 08:21:04.352886 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352869 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 08:21:04.352968 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.352901 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 08:21:04.356563 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.356495 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 08:21:04.358650 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.358636 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 08:21:04.358714 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.358677 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 08:21:04.359428 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359415 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 08:21:04.359493 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359444 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 08:21:04.359493 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359454 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 08:21:04.359493 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359462 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 08:21:04.359493 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359472 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 08:21:04.359493 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359478 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 08:21:04.359630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359604 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 08:21:04.359630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359610 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 08:21:04.359630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359617 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 08:21:04.359630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359622 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 08:21:04.359630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359630 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 08:21:04.359755 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359638 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 08:21:04.359755 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359661 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 08:21:04.359755 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.359677 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 08:21:04.362655 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.362642 2580 server.go:1268] "Started kubelet" Jul 29 08:21:04.362955 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.362776 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 08:21:04.362955 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.362874 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 08:21:04.363424 ip-10-0-138-229 systemd[1]: Started Kubernetes Kubelet. Jul 29 08:21:04.363979 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.363731 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 08:21:04.365321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.365303 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 08:21:04.365400 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.365296 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 08:21:04.365400 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.365352 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 08:21:04.366604 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.366589 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 29 08:21:04.367219 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.367202 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 08:21:04.368377 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.368358 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 08:21:04.370527 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.369426 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-229.ec2.internal.18c6b4f477206d05 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-229.ec2.internal,UID:ip-10-0-138-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-229.ec2.internal,},FirstTimestamp:2026-07-29 08:21:04.362614021 +0000 UTC m=+0.314876831,LastTimestamp:2026-07-29 08:21:04.362614021 +0000 UTC m=+0.314876831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-229.ec2.internal,}" Jul 29 08:21:04.377711 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.377692 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 08:21:04.377795 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.377710 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 08:21:04.377945 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.377925 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:04.378474 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.378450 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 08:21:04.378474 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.378450 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 08:21:04.378474 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.378468 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 08:21:04.378649 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.378561 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 29 08:21:04.378649 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.378572 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 29 08:21:04.378745 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.378656 2580 factory.go:55] Registering systemd factory Jul 29 08:21:04.378745 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.378672 2580 factory.go:223] Registration of the systemd container factory successfully Jul 29 08:21:04.380300 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.380282 2580 factory.go:153] Registering CRI-O factory Jul 29 08:21:04.380300 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.380302 2580 factory.go:223] Registration of the crio container factory successfully Jul 29 08:21:04.380453 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.380352 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 08:21:04.380453 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.380383 2580 factory.go:103] Registering Raw factory Jul 29 08:21:04.380453 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.380399 2580 manager.go:1196] Started watching for new ooms in manager Jul 29 08:21:04.380952 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.380922 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 08:21:04.380952 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.380944 2580 manager.go:349] Starting recovery of all containers Jul 29 08:21:04.381112 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.381029 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 08:21:04.391145 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.391129 2580 manager.go:354] Recovery completed Jul 29 08:21:04.394981 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.394968 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:04.396437 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.396417 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w49sc" Jul 29 08:21:04.396691 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.396680 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:04.396739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.396704 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:04.396739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.396714 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:04.398535 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.398519 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 08:21:04.398535 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.398534 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 08:21:04.398653 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.398552 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 29 08:21:04.398653 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.398556 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:04.399190 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.399175 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:04.399250 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.399202 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:04.399250 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.399215 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:04.399576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.399565 2580 policy_none.go:49] "None policy: Start" Jul 29 08:21:04.399611 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.399581 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 08:21:04.399611 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.399591 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 08:21:04.400664 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.400653 2580 policy_none.go:47] "Start" Jul 29 08:21:04.403725 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.403582 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w49sc" Jul 29 08:21:04.405194 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.405133 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-229.ec2.internal.18c6b4f4792869bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-229.ec2.internal,UID:ip-10-0-138-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-229.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-229.ec2.internal,},FirstTimestamp:2026-07-29 08:21:04.396691901 +0000 UTC m=+0.348954711,LastTimestamp:2026-07-29 08:21:04.396691901 +0000 UTC m=+0.348954711,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-229.ec2.internal,}" Jul 29 08:21:04.408289 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.408234 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-229.ec2.internal.18c6b4f47928aad4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-229.ec2.internal,UID:ip-10-0-138-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-229.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-229.ec2.internal,},FirstTimestamp:2026-07-29 08:21:04.396708564 +0000 UTC m=+0.348971374,LastTimestamp:2026-07-29 08:21:04.396708564 +0000 UTC m=+0.348971374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-229.ec2.internal,}" Jul 29 08:21:04.440430 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.440416 2580 manager.go:337] "Starting Device Plugin manager" Jul 29 08:21:04.440495 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.440447 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 08:21:04.440495 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.440457 2580 server.go:87] "Starting device plugin registration server" Jul 29 08:21:04.440692 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.440676 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 08:21:04.440784 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.440701 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 08:21:04.441173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.440870 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 08:21:04.441173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.440951 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 08:21:04.441173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.440961 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 08:21:04.441430 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.441395 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 08:21:04.441502 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.441445 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:04.461035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.461013 2580 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2613 into the system container \"/system.slice\": " Jul 29 08:21:04.520905 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.520879 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 08:21:04.522180 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.522164 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 08:21:04.522238 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.522213 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 08:21:04.522276 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.522240 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 08:21:04.522317 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.522281 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 08:21:04.524670 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.524651 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:21:04.541499 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.541461 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:04.542366 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.542348 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:04.542463 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.542379 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:04.542463 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.542393 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:04.542463 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.542421 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.552451 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.552436 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.552520 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.552456 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-229.ec2.internal\": node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:04.579762 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.579741 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:04.623305 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.623288 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal"] Jul 29 08:21:04.623389 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.623340 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:04.624784 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.624772 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:04.624867 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.624796 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:04.624867 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.624805 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:04.625924 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.625912 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:04.626035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.626070 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626049 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:04.626666 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626651 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:04.626720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626654 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:04.626720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626681 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:04.626720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626695 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:04.626720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626697 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:04.626720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.626711 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:04.627899 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.627882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.627942 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.627916 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:04.628558 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.628543 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:04.628722 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.628568 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:04.628722 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.628580 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:04.649494 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.649470 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-229.ec2.internal\" not found" node="ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.653883 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.653869 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-229.ec2.internal\" not found" node="ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.679824 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.679796 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:04.679912 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.679866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f58e40be53808051ab96b997926e90bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal\" (UID: \"f58e40be53808051ab96b997926e90bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.679912 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.679886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f58e40be53808051ab96b997926e90bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal\" (UID: \"f58e40be53808051ab96b997926e90bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.679912 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.679901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39e627514db70702d43d1afa7e7236cb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-229.ec2.internal\" (UID: \"39e627514db70702d43d1afa7e7236cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.780216 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.780195 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:04.780285 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.780237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f58e40be53808051ab96b997926e90bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal\" (UID: \"f58e40be53808051ab96b997926e90bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.780322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.780283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f58e40be53808051ab96b997926e90bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal\" (UID: \"f58e40be53808051ab96b997926e90bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.780322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.780257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f58e40be53808051ab96b997926e90bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal\" (UID: \"f58e40be53808051ab96b997926e90bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.780382 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.780322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39e627514db70702d43d1afa7e7236cb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-229.ec2.internal\" (UID: \"39e627514db70702d43d1afa7e7236cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.780382 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.780322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f58e40be53808051ab96b997926e90bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal\" (UID: \"f58e40be53808051ab96b997926e90bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.780382 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.780346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39e627514db70702d43d1afa7e7236cb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-229.ec2.internal\" (UID: \"39e627514db70702d43d1afa7e7236cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.880810 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.880769 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:04.951965 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.951945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.958261 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:04.958246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" Jul 29 08:21:04.980963 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:04.980939 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:05.081420 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:05.081396 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:05.181784 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:05.181739 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:05.282109 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:05.282088 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:05.313272 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.313244 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 08:21:05.313646 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.313407 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 08:21:05.383016 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:05.383000 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:05.408382 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.408355 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 08:16:05 +0000 UTC" deadline="2028-04-06 06:59:41.785106096 +0000 UTC" Jul 29 08:21:05.408382 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.408381 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14806h38m36.376728136s" Jul 29 08:21:05.468869 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:05.468820 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58e40be53808051ab96b997926e90bd.slice/crio-bd97a00acfa400281e920ca51744c2460037cb3b115a4facfb3e105c9e119575 WatchSource:0}: Error finding container bd97a00acfa400281e920ca51744c2460037cb3b115a4facfb3e105c9e119575: Status 404 returned error can't find the container with id bd97a00acfa400281e920ca51744c2460037cb3b115a4facfb3e105c9e119575 Jul 29 08:21:05.469226 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:05.469207 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e627514db70702d43d1afa7e7236cb.slice/crio-67c836d2fdf03d0378fab71d30ec54762b65ad7b68705dbaf1d87212f33259dd WatchSource:0}: Error finding container 67c836d2fdf03d0378fab71d30ec54762b65ad7b68705dbaf1d87212f33259dd: Status 404 returned error can't find the container with id 67c836d2fdf03d0378fab71d30ec54762b65ad7b68705dbaf1d87212f33259dd Jul 29 08:21:05.474102 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.474080 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:21:05.483747 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:05.483726 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:05.526163 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.526127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" event={"ID":"f58e40be53808051ab96b997926e90bd","Type":"ContainerStarted","Data":"bd97a00acfa400281e920ca51744c2460037cb3b115a4facfb3e105c9e119575"} Jul 29 08:21:05.527035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.527015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" event={"ID":"39e627514db70702d43d1afa7e7236cb","Type":"ContainerStarted","Data":"67c836d2fdf03d0378fab71d30ec54762b65ad7b68705dbaf1d87212f33259dd"} Jul 29 08:21:05.535224 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.535212 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:21:05.536146 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.536132 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:21:05.536207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.536158 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:21:05.536207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.536170 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:21:05.583898 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:05.583878 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-229.ec2.internal\" not found" Jul 29 08:21:05.609299 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.609279 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:21:05.678136 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.678117 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" Jul 29 08:21:05.693519 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.693503 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 08:21:05.695055 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.695043 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" Jul 29 08:21:05.705161 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.705143 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 08:21:05.861618 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.861596 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:21:05.866213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:05.866191 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:21:06.133752 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.133675 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:21:06.354530 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.354502 2580 apiserver.go:52] "Watching apiserver" Jul 29 08:21:06.361777 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.361756 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 08:21:06.363483 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.363462 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-fztml","kube-system/konnectivity-agent-ssfvp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr","openshift-cluster-node-tuning-operator/tuned-m7c5r","openshift-image-registry/node-ca-w5j4b","openshift-multus/multus-svvdf","openshift-network-operator/iptables-alerter-pb88v","kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal","openshift-dns/node-resolver-lw44r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal","openshift-multus/multus-additional-cni-plugins-p2w7k","openshift-multus/network-metrics-daemon-vmj9l","openshift-network-diagnostics/network-check-target-hqzk8","openshift-ovn-kubernetes/ovnkube-node-ffbdj"] Jul 29 08:21:06.364763 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.364743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.366703 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.366682 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:06.367429 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.367406 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 08:21:06.367520 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.367412 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 08:21:06.367581 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.367570 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 08:21:06.367883 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.367866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:06.368293 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.368272 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5h44f\"" Jul 29 08:21:06.369073 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.369047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 08:21:06.369073 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.369060 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjq2g\"" Jul 29 08:21:06.369183 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.369129 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 08:21:06.369367 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.369354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.369839 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.369820 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:21:06.370277 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.370256 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 08:21:06.370688 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.370265 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9pn9p\"" Jul 29 08:21:06.370688 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.370401 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 08:21:06.370688 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.370407 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:06.371759 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.371741 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:21:06.372781 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.372323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.373109 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.373090 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qvmpd\"" Jul 29 08:21:06.373234 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.373217 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 08:21:06.373292 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.373228 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 08:21:06.373338 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.373325 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 08:21:06.373486 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.373471 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:21:06.374662 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.373898 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mjwqc\"" Jul 29 08:21:06.374662 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.373934 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:06.374662 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:06.374056 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:06.376384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.375337 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 08:21:06.376384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.375406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:06.376384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.376035 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 08:21:06.376384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.376063 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 08:21:06.376384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.376198 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 08:21:06.376384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.376256 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q5vtr\"" Jul 29 08:21:06.377195 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.377174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.377525 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.377510 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 08:21:06.377702 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.377675 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-slwh4\"" Jul 29 08:21:06.377794 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.377717 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 08:21:06.378080 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.378063 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 08:21:06.378439 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.378423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:06.378505 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:06.378485 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:06.379382 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.379365 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 08:21:06.379526 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.379509 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2fhbh\"" Jul 29 08:21:06.379684 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.379670 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 08:21:06.380125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.380107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:06.380199 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:06.380179 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:06.381450 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.381431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:06.383782 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.383735 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 08:21:06.383980 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.383962 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 08:21:06.384062 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.384003 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6dk2v\"" Jul 29 08:21:06.384227 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.384210 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 08:21:06.384310 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.384233 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 08:21:06.384368 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.384315 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 08:21:06.384368 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.384326 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 08:21:06.387757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3136e-5a0b-4c82-a3b4-5d43b35d442e-konnectivity-ca\") pod \"konnectivity-agent-ssfvp\" (UID: \"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e\") " pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:06.387829 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-os-release\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.387829 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cni-binary-copy\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.387829 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysctl-conf\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.387829 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-systemd\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.387993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13fd06f4-1b82-4049-b980-0ef9d1ae0742-iptables-alerter-script\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:06.387993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysctl-d\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.387993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-multus-certs\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.387993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-device-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.387993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-system-cni-dir\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.387993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.387987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-sys\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-cni-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkcrl\" (UniqueName: \"kubernetes.io/projected/bf109907-5eb4-404b-8454-e87a1e2c5849-kube-api-access-tkcrl\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/928e848d-cc8f-4abe-b018-ea9084a8f303-cni-binary-copy\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-k8s-cni-cncf-io\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62056413-e3c1-4562-b6ae-6ee8a49df104-dbus\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0006c3a5-faa2-4b14-a57c-1e1b468041a8-host\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-daemon-config\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-etc-kubernetes\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-socket-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-sys-fs\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cnibin\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62056413-e3c1-4562-b6ae-6ee8a49df104-kubelet-config\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5beacbc7-a61e-4621-912e-d3807d189d2d-hosts-file\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7e3136e-5a0b-4c82-a3b4-5d43b35d442e-agent-certs\") pod \"konnectivity-agent-ssfvp\" (UID: \"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e\") " pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13fd06f4-1b82-4049-b980-0ef9d1ae0742-host-slash\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beacbc7-a61e-4621-912e-d3807d189d2d-tmp-dir\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-var-lib-kubelet\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40efe082-36d8-470a-9c42-754292673df6-etc-tuned\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-cnibin\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-socket-dir-parent\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-hostroot\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.388567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdw2\" (UniqueName: \"kubernetes.io/projected/928e848d-cc8f-4abe-b018-ea9084a8f303-kube-api-access-ksdw2\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6txk\" (UniqueName: \"kubernetes.io/projected/5beacbc7-a61e-4621-912e-d3807d189d2d-kube-api-access-z6txk\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-modprobe-d\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-lib-modules\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-host\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm2lt\" (UniqueName: \"kubernetes.io/projected/40efe082-36d8-470a-9c42-754292673df6-kube-api-access-qm2lt\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-cni-bin\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7sp\" (UniqueName: \"kubernetes.io/projected/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-kube-api-access-cw7sp\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-cni-multus\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-conf-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-registration-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wnn7\" (UniqueName: \"kubernetes.io/projected/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-kube-api-access-5wnn7\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-run\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389090 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.388988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p87\" (UniqueName: \"kubernetes.io/projected/0006c3a5-faa2-4b14-a57c-1e1b468041a8-kube-api-access-h2p87\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-system-cni-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-os-release\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-netns\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-kubelet\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389121 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmzk\" (UniqueName: \"kubernetes.io/projected/13fd06f4-1b82-4049-b980-0ef9d1ae0742-kube-api-access-rtmzk\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysconfig\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-kubernetes\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0006c3a5-faa2-4b14-a57c-1e1b468041a8-serviceca\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40efe082-36d8-470a-9c42-754292673df6-tmp\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.389705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.389355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:06.391369 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.391349 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 08:21:06.418720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.418703 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ddmnl" Jul 29 08:21:06.429276 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.428806 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ddmnl" Jul 29 08:21:06.480097 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:06.480071 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 08:21:07.863089 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13fd06f4-1b82-4049-b980-0ef9d1ae0742-host-slash\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:07.863187 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-log-socket\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.863187 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.863187 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beacbc7-a61e-4621-912e-d3807d189d2d-tmp-dir\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:07.863343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-var-lib-kubelet\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.863343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13fd06f4-1b82-4049-b980-0ef9d1ae0742-host-slash\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:07.863343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40efe082-36d8-470a-9c42-754292673df6-etc-tuned\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.863343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863267 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-var-lib-kubelet\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.863343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-systemd-units\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.863343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-cnibin\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-socket-dir-parent\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-hostroot\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdw2\" (UniqueName: \"kubernetes.io/projected/928e848d-cc8f-4abe-b018-ea9084a8f303-kube-api-access-ksdw2\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6txk\" (UniqueName: \"kubernetes.io/projected/5beacbc7-a61e-4621-912e-d3807d189d2d-kube-api-access-z6txk\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beacbc7-a61e-4621-912e-d3807d189d2d-tmp-dir\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-socket-dir-parent\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-var-lib-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863493 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-ovnkube-config\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-cnibin\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-modprobe-d\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-lib-modules\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-host\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.863632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2lt\" (UniqueName: \"kubernetes.io/projected/40efe082-36d8-470a-9c42-754292673df6-kube-api-access-qm2lt\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-run-netns\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-cni-bin\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-cni-bin\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-host\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863781 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-lib-modules\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863786 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-hostroot\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.863877 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863886 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-modprobe-d\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.863923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7sp\" (UniqueName: \"kubernetes.io/projected/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-kube-api-access-cw7sp\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.863942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:08.363923022 +0000 UTC m=+2.943604716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.864101 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.864161 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret podName:62056413-e3c1-4562-b6ae-6ee8a49df104 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:08.364146401 +0000 UTC m=+2.943828069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret") pod "global-pull-secret-syncer-fztml" (UID: "62056413-e3c1-4562-b6ae-6ee8a49df104") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-cni-multus\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-conf-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.864328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-registration-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wnn7\" (UniqueName: \"kubernetes.io/projected/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-kube-api-access-5wnn7\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-cni-multus\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-conf-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-run\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p87\" (UniqueName: \"kubernetes.io/projected/0006c3a5-faa2-4b14-a57c-1e1b468041a8-kube-api-access-h2p87\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-run\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-env-overrides\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-system-cni-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-os-release\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-system-cni-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-netns\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-kubelet\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-os-release\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmzk\" (UniqueName: \"kubernetes.io/projected/13fd06f4-1b82-4049-b980-0ef9d1ae0742-kube-api-access-rtmzk\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864756 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-var-lib-kubelet\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysconfig\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-kubernetes\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-netns\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysconfig\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0006c3a5-faa2-4b14-a57c-1e1b468041a8-serviceca\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-kubernetes\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.864973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40efe082-36d8-470a-9c42-754292673df6-tmp\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-slash\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-cni-netd\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3136e-5a0b-4c82-a3b4-5d43b35d442e-konnectivity-ca\") pod \"konnectivity-agent-ssfvp\" (UID: \"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e\") " pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-os-release\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cni-binary-copy\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysctl-conf\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.865769 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0006c3a5-faa2-4b14-a57c-1e1b468041a8-serviceca\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-systemd\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-systemd\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13fd06f4-1b82-4049-b980-0ef9d1ae0742-iptables-alerter-script\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysctl-conf\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysctl-d\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-os-release\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-node-log\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-ovnkube-script-lib\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-multus-certs\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-device-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-system-cni-dir\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3136e-5a0b-4c82-a3b4-5d43b35d442e-konnectivity-ca\") pod \"konnectivity-agent-ssfvp\" (UID: \"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e\") " pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-multus-certs\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-etc-sysctl-d\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.866591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-device-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-system-cni-dir\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-sys\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865892 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cni-binary-copy\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-cni-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40efe082-36d8-470a-9c42-754292673df6-sys\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.865971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-cni-bin\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcrl\" (UniqueName: \"kubernetes.io/projected/bf109907-5eb4-404b-8454-e87a1e2c5849-kube-api-access-tkcrl\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-systemd\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866063 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-cni-dir\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdpj\" (UniqueName: \"kubernetes.io/projected/222f904f-3dff-40ac-a735-40260879417f-kube-api-access-xxdpj\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13fd06f4-1b82-4049-b980-0ef9d1ae0742-iptables-alerter-script\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/928e848d-cc8f-4abe-b018-ea9084a8f303-cni-binary-copy\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-k8s-cni-cncf-io\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62056413-e3c1-4562-b6ae-6ee8a49df104-dbus\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:07.867347 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-host-run-k8s-cni-cncf-io\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62056413-e3c1-4562-b6ae-6ee8a49df104-dbus\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0006c3a5-faa2-4b14-a57c-1e1b468041a8-host\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-kubelet\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-etc-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0006c3a5-faa2-4b14-a57c-1e1b468041a8-host\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-daemon-config\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-etc-kubernetes\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-socket-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/928e848d-cc8f-4abe-b018-ea9084a8f303-etc-kubernetes\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-sys-fs\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-registration-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cnibin\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/928e848d-cc8f-4abe-b018-ea9084a8f303-cni-binary-copy\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-sys-fs\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.868004 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-socket-dir\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-ovn\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866751 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-cnibin\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/222f904f-3dff-40ac-a735-40260879417f-ovn-node-metrics-cert\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62056413-e3c1-4562-b6ae-6ee8a49df104-kubelet-config\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5beacbc7-a61e-4621-912e-d3807d189d2d-hosts-file\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7e3136e-5a0b-4c82-a3b4-5d43b35d442e-agent-certs\") pod \"konnectivity-agent-ssfvp\" (UID: \"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e\") " pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.866984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5beacbc7-a61e-4621-912e-d3807d189d2d-hosts-file\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.867002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/928e848d-cc8f-4abe-b018-ea9084a8f303-multus-daemon-config\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.867055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62056413-e3c1-4562-b6ae-6ee8a49df104-kubelet-config\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.867285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40efe082-36d8-470a-9c42-754292673df6-etc-tuned\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.868573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.867578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40efe082-36d8-470a-9c42-754292673df6-tmp\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.869274 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.869252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7e3136e-5a0b-4c82-a3b4-5d43b35d442e-agent-certs\") pod \"konnectivity-agent-ssfvp\" (UID: \"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e\") " pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:07.879599 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.879579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdw2\" (UniqueName: \"kubernetes.io/projected/928e848d-cc8f-4abe-b018-ea9084a8f303-kube-api-access-ksdw2\") pod \"multus-svvdf\" (UID: \"928e848d-cc8f-4abe-b018-ea9084a8f303\") " pod="openshift-multus/multus-svvdf" Jul 29 08:21:07.880158 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.880138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wnn7\" (UniqueName: \"kubernetes.io/projected/3be7fa5e-32eb-4151-9f4a-9b80f2420d72-kube-api-access-5wnn7\") pod \"aws-ebs-csi-driver-node-s6mxr\" (UID: \"3be7fa5e-32eb-4151-9f4a-9b80f2420d72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:07.881133 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.881114 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:21:07.881226 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.881139 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:21:07.881226 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.881152 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7v4fb for pod openshift-network-diagnostics/network-check-target-hqzk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:07.881226 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:07.881206 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb podName:a5014e06-bb11-4b83-854d-74f033f51cb9 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:08.381189674 +0000 UTC m=+2.960871348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7v4fb" (UniqueName: "kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb") pod "network-check-target-hqzk8" (UID: "a5014e06-bb11-4b83-854d-74f033f51cb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:07.882641 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.882597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcrl\" (UniqueName: \"kubernetes.io/projected/bf109907-5eb4-404b-8454-e87a1e2c5849-kube-api-access-tkcrl\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:07.882724 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.882684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p87\" (UniqueName: \"kubernetes.io/projected/0006c3a5-faa2-4b14-a57c-1e1b468041a8-kube-api-access-h2p87\") pod \"node-ca-w5j4b\" (UID: \"0006c3a5-faa2-4b14-a57c-1e1b468041a8\") " pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:07.882724 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.882687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2lt\" (UniqueName: \"kubernetes.io/projected/40efe082-36d8-470a-9c42-754292673df6-kube-api-access-qm2lt\") pod \"tuned-m7c5r\" (UID: \"40efe082-36d8-470a-9c42-754292673df6\") " pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:07.883007 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.882988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7sp\" (UniqueName: \"kubernetes.io/projected/5ebeefc4-a87d-46db-8f40-d2cebb70eca1-kube-api-access-cw7sp\") pod \"multus-additional-cni-plugins-p2w7k\" (UID: \"5ebeefc4-a87d-46db-8f40-d2cebb70eca1\") " pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:07.884550 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.884514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmzk\" (UniqueName: \"kubernetes.io/projected/13fd06f4-1b82-4049-b980-0ef9d1ae0742-kube-api-access-rtmzk\") pod \"iptables-alerter-pb88v\" (UID: \"13fd06f4-1b82-4049-b980-0ef9d1ae0742\") " pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:07.884620 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.884546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6txk\" (UniqueName: \"kubernetes.io/projected/5beacbc7-a61e-4621-912e-d3807d189d2d-kube-api-access-z6txk\") pod \"node-resolver-lw44r\" (UID: \"5beacbc7-a61e-4621-912e-d3807d189d2d\") " pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:07.967845 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-ovn\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/222f904f-3dff-40ac-a735-40260879417f-ovn-node-metrics-cert\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-log-socket\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-systemd-units\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-var-lib-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-ovn\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-ovnkube-config\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.967987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-systemd-units\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-var-lib-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-run-netns\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968039 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-log-socket\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-env-overrides\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-run-netns\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-slash\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-cni-netd\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-node-log\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-ovnkube-script-lib\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-cni-netd\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-slash\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-node-log\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-cni-bin\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-systemd\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-cni-bin\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.968654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdpj\" (UniqueName: \"kubernetes.io/projected/222f904f-3dff-40ac-a735-40260879417f-kube-api-access-xxdpj\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-kubelet\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-etc-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-env-overrides\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968555 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-etc-openvswitch\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-host-kubelet\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-ovnkube-config\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/222f904f-3dff-40ac-a735-40260879417f-run-systemd\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.969377 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.968737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/222f904f-3dff-40ac-a735-40260879417f-ovnkube-script-lib\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.970676 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.970653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/222f904f-3dff-40ac-a735-40260879417f-ovn-node-metrics-cert\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:07.980583 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:07.980561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdpj\" (UniqueName: \"kubernetes.io/projected/222f904f-3dff-40ac-a735-40260879417f-kube-api-access-xxdpj\") pod \"ovnkube-node-ffbdj\" (UID: \"222f904f-3dff-40ac-a735-40260879417f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:08.053221 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.053151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" Jul 29 08:21:08.058207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.058184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lw44r" Jul 29 08:21:08.068626 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.068606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pb88v" Jul 29 08:21:08.072622 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.072606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" Jul 29 08:21:08.079217 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.079200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w5j4b" Jul 29 08:21:08.085752 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.085736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-svvdf" Jul 29 08:21:08.093202 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.093183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:08.098592 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.098576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" Jul 29 08:21:08.104144 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.104120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:08.371031 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.370956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:08.371031 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.370993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:08.371194 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.371099 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:08.371194 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.371100 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:08.371194 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.371154 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret podName:62056413-e3c1-4562-b6ae-6ee8a49df104 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:09.371140679 +0000 UTC m=+3.950822343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret") pod "global-pull-secret-syncer-fztml" (UID: "62056413-e3c1-4562-b6ae-6ee8a49df104") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:08.371194 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.371168 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:09.371161899 +0000 UTC m=+3.950843560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:08.471489 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.471452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:08.471655 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.471633 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:21:08.471705 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.471662 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:21:08.471705 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.471672 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7v4fb for pod openshift-network-diagnostics/network-check-target-hqzk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:08.471773 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.471723 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb podName:a5014e06-bb11-4b83-854d-74f033f51cb9 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:09.471709921 +0000 UTC m=+4.051391582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7v4fb" (UniqueName: "kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb") pod "network-check-target-hqzk8" (UID: "a5014e06-bb11-4b83-854d-74f033f51cb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:08.706337 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.706233 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40efe082_36d8_470a_9c42_754292673df6.slice/crio-dc8b3354dc835a12baf756cf8ce1771442831f3809bee89f0a2f79ec9bceaf63 WatchSource:0}: Error finding container dc8b3354dc835a12baf756cf8ce1771442831f3809bee89f0a2f79ec9bceaf63: Status 404 returned error can't find the container with id dc8b3354dc835a12baf756cf8ce1771442831f3809bee89f0a2f79ec9bceaf63 Jul 29 08:21:08.709572 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.709233 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fd06f4_1b82_4049_b980_0ef9d1ae0742.slice/crio-f51c248d2a6ca6ffb51119658f051c5454b2e885430b3346d504e6f6391636d4 WatchSource:0}: Error finding container f51c248d2a6ca6ffb51119658f051c5454b2e885430b3346d504e6f6391636d4: Status 404 returned error can't find the container with id f51c248d2a6ca6ffb51119658f051c5454b2e885430b3346d504e6f6391636d4 Jul 29 08:21:08.713726 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.713695 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222f904f_3dff_40ac_a735_40260879417f.slice/crio-46f26c8d5a04c7ce372c56e420b44498b746b8f1cce31967219064c126e7e857 WatchSource:0}: Error finding container 46f26c8d5a04c7ce372c56e420b44498b746b8f1cce31967219064c126e7e857: Status 404 returned error can't find the container with id 46f26c8d5a04c7ce372c56e420b44498b746b8f1cce31967219064c126e7e857 Jul 29 08:21:08.714225 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.714199 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5beacbc7_a61e_4621_912e_d3807d189d2d.slice/crio-b679c796e58ea9787e7620b4521c369d3aab315ba291fd08da07db25ac53438e WatchSource:0}: Error finding container b679c796e58ea9787e7620b4521c369d3aab315ba291fd08da07db25ac53438e: Status 404 returned error can't find the container with id b679c796e58ea9787e7620b4521c369d3aab315ba291fd08da07db25ac53438e Jul 29 08:21:08.715992 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.715975 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e3136e_5a0b_4c82_a3b4_5d43b35d442e.slice/crio-8ae18a5fabe5b65de5c63269c88a8208d1e44d28af6b89c1088985b028083a1e WatchSource:0}: Error finding container 8ae18a5fabe5b65de5c63269c88a8208d1e44d28af6b89c1088985b028083a1e: Status 404 returned error can't find the container with id 8ae18a5fabe5b65de5c63269c88a8208d1e44d28af6b89c1088985b028083a1e Jul 29 08:21:08.717068 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.717033 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928e848d_cc8f_4abe_b018_ea9084a8f303.slice/crio-4ad4aa9e1209bfc879bfa133b8957e35c7819ce3da99a95075277e208c5f06b1 WatchSource:0}: Error finding container 4ad4aa9e1209bfc879bfa133b8957e35c7819ce3da99a95075277e208c5f06b1: Status 404 returned error can't find the container with id 4ad4aa9e1209bfc879bfa133b8957e35c7819ce3da99a95075277e208c5f06b1 Jul 29 08:21:08.718001 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.717979 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebeefc4_a87d_46db_8f40_d2cebb70eca1.slice/crio-8ba4ce08ee2cf8ca9817e0224b1456f8ca52aa654064f18e67d47043cd9e37e9 WatchSource:0}: Error finding container 8ba4ce08ee2cf8ca9817e0224b1456f8ca52aa654064f18e67d47043cd9e37e9: Status 404 returned error can't find the container with id 8ba4ce08ee2cf8ca9817e0224b1456f8ca52aa654064f18e67d47043cd9e37e9 Jul 29 08:21:08.719684 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.719660 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0006c3a5_faa2_4b14_a57c_1e1b468041a8.slice/crio-510f4f9cbdf8b49987632178bdef992ca99676bf1b37687fbd750f53cc0d4f3a WatchSource:0}: Error finding container 510f4f9cbdf8b49987632178bdef992ca99676bf1b37687fbd750f53cc0d4f3a: Status 404 returned error can't find the container with id 510f4f9cbdf8b49987632178bdef992ca99676bf1b37687fbd750f53cc0d4f3a Jul 29 08:21:08.721098 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:08.721077 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be7fa5e_32eb_4151_9f4a_9b80f2420d72.slice/crio-cf69b8e604e5633169d71a0e09dbbe0ccc51f4381103087033fc885d2cd9c359 WatchSource:0}: Error finding container cf69b8e604e5633169d71a0e09dbbe0ccc51f4381103087033fc885d2cd9c359: Status 404 returned error can't find the container with id cf69b8e604e5633169d71a0e09dbbe0ccc51f4381103087033fc885d2cd9c359 Jul 29 08:21:08.805464 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.805406 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 08:16:07 +0000 UTC" deadline="2028-03-03 04:46:34.828925212 +0000 UTC" Jul 29 08:21:08.805464 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.805447 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13988h25m26.023481512s" Jul 29 08:21:08.895536 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.895499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:08.895698 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.895630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:08.895859 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.895499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:08.895963 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:08.895941 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:08.902879 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.902856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" event={"ID":"3be7fa5e-32eb-4151-9f4a-9b80f2420d72","Type":"ContainerStarted","Data":"cf69b8e604e5633169d71a0e09dbbe0ccc51f4381103087033fc885d2cd9c359"} Jul 29 08:21:08.903835 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.903817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lw44r" event={"ID":"5beacbc7-a61e-4621-912e-d3807d189d2d","Type":"ContainerStarted","Data":"b679c796e58ea9787e7620b4521c369d3aab315ba291fd08da07db25ac53438e"} Jul 29 08:21:08.904848 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.904829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w5j4b" event={"ID":"0006c3a5-faa2-4b14-a57c-1e1b468041a8","Type":"ContainerStarted","Data":"510f4f9cbdf8b49987632178bdef992ca99676bf1b37687fbd750f53cc0d4f3a"} Jul 29 08:21:08.905753 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.905735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-svvdf" event={"ID":"928e848d-cc8f-4abe-b018-ea9084a8f303","Type":"ContainerStarted","Data":"4ad4aa9e1209bfc879bfa133b8957e35c7819ce3da99a95075277e208c5f06b1"} Jul 29 08:21:08.906654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.906636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ssfvp" event={"ID":"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e","Type":"ContainerStarted","Data":"8ae18a5fabe5b65de5c63269c88a8208d1e44d28af6b89c1088985b028083a1e"} Jul 29 08:21:08.907710 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.907690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"46f26c8d5a04c7ce372c56e420b44498b746b8f1cce31967219064c126e7e857"} Jul 29 08:21:08.908667 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.908645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pb88v" event={"ID":"13fd06f4-1b82-4049-b980-0ef9d1ae0742","Type":"ContainerStarted","Data":"f51c248d2a6ca6ffb51119658f051c5454b2e885430b3346d504e6f6391636d4"} Jul 29 08:21:08.912321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.912292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerStarted","Data":"8ba4ce08ee2cf8ca9817e0224b1456f8ca52aa654064f18e67d47043cd9e37e9"} Jul 29 08:21:08.913244 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.913227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" event={"ID":"40efe082-36d8-470a-9c42-754292673df6","Type":"ContainerStarted","Data":"dc8b3354dc835a12baf756cf8ce1771442831f3809bee89f0a2f79ec9bceaf63"} Jul 29 08:21:08.915328 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:08.915304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" event={"ID":"39e627514db70702d43d1afa7e7236cb","Type":"ContainerStarted","Data":"2f754c574d3510836e00baeea8c3a04d7c453b85df7f13a6ac0729f248b43e6a"} Jul 29 08:21:09.379562 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.379480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:09.379562 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.379534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:09.379783 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.379684 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:09.379783 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.379745 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret podName:62056413-e3c1-4562-b6ae-6ee8a49df104 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:11.379728684 +0000 UTC m=+5.959410358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret") pod "global-pull-secret-syncer-fztml" (UID: "62056413-e3c1-4562-b6ae-6ee8a49df104") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:09.380138 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.380116 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:09.380225 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.380170 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:11.380154617 +0000 UTC m=+5.959836287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:09.481143 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.480523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:09.481143 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.480735 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:21:09.481143 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.480754 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:21:09.481143 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.480768 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7v4fb for pod openshift-network-diagnostics/network-check-target-hqzk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:09.481143 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.480826 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb podName:a5014e06-bb11-4b83-854d-74f033f51cb9 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:11.480808354 +0000 UTC m=+6.060490037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7v4fb" (UniqueName: "kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb") pod "network-check-target-hqzk8" (UID: "a5014e06-bb11-4b83-854d-74f033f51cb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:09.806407 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.806339 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 08:16:07 +0000 UTC" deadline="2028-03-08 21:40:36.299036616 +0000 UTC" Jul 29 08:21:09.806407 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.806405 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14125h19m26.492636409s" Jul 29 08:21:09.898145 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.898120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:09.898274 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:09.898248 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:09.933213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.933180 2580 generic.go:356] "Generic (PLEG): container finished" podID="f58e40be53808051ab96b997926e90bd" containerID="fdc1bc67deecae0dbfbee845fdabe12a659d379f0a1cd3c048d22268fae52977" exitCode=0 Jul 29 08:21:09.933539 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.933507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" event={"ID":"f58e40be53808051ab96b997926e90bd","Type":"ContainerDied","Data":"fdc1bc67deecae0dbfbee845fdabe12a659d379f0a1cd3c048d22268fae52977"} Jul 29 08:21:09.954340 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:09.954157 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-229.ec2.internal" podStartSLOduration=2.954136308 podStartE2EDuration="2.954136308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:08.937624202 +0000 UTC m=+3.517305886" watchObservedRunningTime="2026-07-29 08:21:09.954136308 +0000 UTC m=+4.533817993" Jul 29 08:21:10.895859 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:10.895834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:10.896350 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:10.895871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:10.896350 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:10.895974 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:10.896350 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:10.896193 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:10.957734 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:10.957692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" event={"ID":"f58e40be53808051ab96b997926e90bd","Type":"ContainerStarted","Data":"0c4b4b7c025c382ae2964c63b7364b6f305dc6a5df19fde161319f750ce5f9cb"} Jul 29 08:21:10.976958 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:10.976075 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-229.ec2.internal" podStartSLOduration=3.97605832 podStartE2EDuration="3.97605832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:10.975689065 +0000 UTC m=+5.555370748" watchObservedRunningTime="2026-07-29 08:21:10.97605832 +0000 UTC m=+5.555740006" Jul 29 08:21:11.397239 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:11.397184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:11.397239 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:11.397231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:11.397511 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.397346 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:11.397511 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.397408 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret podName:62056413-e3c1-4562-b6ae-6ee8a49df104 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:15.397390581 +0000 UTC m=+9.977072244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret") pod "global-pull-secret-syncer-fztml" (UID: "62056413-e3c1-4562-b6ae-6ee8a49df104") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:11.397834 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.397813 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:11.397913 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.397868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:15.397854096 +0000 UTC m=+9.977535763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:11.498167 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:11.498129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:11.498411 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.498390 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:21:11.498532 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.498420 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:21:11.498532 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.498450 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7v4fb for pod openshift-network-diagnostics/network-check-target-hqzk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:11.498532 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.498510 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb podName:a5014e06-bb11-4b83-854d-74f033f51cb9 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:15.498491021 +0000 UTC m=+10.078172688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7v4fb" (UniqueName: "kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb") pod "network-check-target-hqzk8" (UID: "a5014e06-bb11-4b83-854d-74f033f51cb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:11.896667 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:11.896633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:11.897095 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:11.896772 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:12.895834 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:12.895801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:12.895991 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:12.895929 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:12.896038 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:12.895999 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:12.896100 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:12.896081 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:13.896005 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:13.895972 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:13.896461 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:13.896108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:14.896084 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:14.895995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:14.896569 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:14.896127 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:14.896569 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:14.896530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:14.896681 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:14.896646 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:15.431693 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:15.431465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:15.431693 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:15.431509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:15.431693 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.431615 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:15.431693 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.431658 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret podName:62056413-e3c1-4562-b6ae-6ee8a49df104 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:23.43164533 +0000 UTC m=+18.011326991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret") pod "global-pull-secret-syncer-fztml" (UID: "62056413-e3c1-4562-b6ae-6ee8a49df104") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:15.431978 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.431962 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:15.432024 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.432011 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:23.431995523 +0000 UTC m=+18.011677192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:15.532098 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:15.532067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:15.532269 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.532223 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:21:15.532269 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.532243 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:21:15.532269 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.532252 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7v4fb for pod openshift-network-diagnostics/network-check-target-hqzk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:15.532388 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.532289 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb podName:a5014e06-bb11-4b83-854d-74f033f51cb9 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:23.532278491 +0000 UTC m=+18.111960154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7v4fb" (UniqueName: "kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb") pod "network-check-target-hqzk8" (UID: "a5014e06-bb11-4b83-854d-74f033f51cb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:15.895954 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:15.895876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:15.896097 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:15.895976 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:16.895089 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:16.895056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:16.895316 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:16.895064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:16.895316 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:16.895182 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:16.895316 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:16.895271 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:17.897524 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:17.897490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:17.897941 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:17.897629 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:18.895527 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:18.895494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:18.895697 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:18.895494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:18.895697 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:18.895605 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:18.895796 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:18.895725 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:19.895851 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:19.895765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:19.896269 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:19.895900 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:20.895445 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:20.895408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:20.895631 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:20.895544 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:20.895631 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:20.895597 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:20.895759 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:20.895725 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:21.896458 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:21.895963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:21.896458 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:21.896099 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:22.895342 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:22.895302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:22.895629 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:22.895448 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:22.895629 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:22.895487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:22.895629 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:22.895592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:23.490707 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:23.490680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:23.491014 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:23.490715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:23.491014 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.490813 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:23.491014 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.490815 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:23.491014 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.490866 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret podName:62056413-e3c1-4562-b6ae-6ee8a49df104 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:39.490852413 +0000 UTC m=+34.070534073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret") pod "global-pull-secret-syncer-fztml" (UID: "62056413-e3c1-4562-b6ae-6ee8a49df104") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:23.491014 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.490880 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:39.490873283 +0000 UTC m=+34.070554944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:23.591695 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:23.591670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:23.591783 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.591772 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:21:23.591829 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.591789 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:21:23.591829 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.591798 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7v4fb for pod openshift-network-diagnostics/network-check-target-hqzk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:23.591900 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.591839 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb podName:a5014e06-bb11-4b83-854d-74f033f51cb9 nodeName:}" failed. No retries permitted until 2026-07-29 08:21:39.591827424 +0000 UTC m=+34.171509084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7v4fb" (UniqueName: "kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb") pod "network-check-target-hqzk8" (UID: "a5014e06-bb11-4b83-854d-74f033f51cb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:23.895616 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:23.895533 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:23.895823 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:23.895674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:24.895278 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:24.895242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:24.895716 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:24.895361 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:24.895716 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:24.895469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:24.895716 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:24.895579 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:25.896640 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:25.896607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:25.897091 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:25.896712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:26.895297 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:26.895266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:26.895477 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:26.895360 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:26.895477 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:26.895423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:26.895596 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:26.895568 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:27.895687 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.895345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:27.896359 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:27.895758 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:27.991912 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.991876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" event={"ID":"3be7fa5e-32eb-4151-9f4a-9b80f2420d72","Type":"ContainerStarted","Data":"98b77e08c239b1e47c92109e71d09220423e5c17f3812e742d347c0669b06560"} Jul 29 08:21:27.993041 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.993019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lw44r" event={"ID":"5beacbc7-a61e-4621-912e-d3807d189d2d","Type":"ContainerStarted","Data":"e0dd25eade8ccb159427b737e95908ac24fa34040ad437aee0b0f954e2562576"} Jul 29 08:21:27.994477 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.994421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w5j4b" event={"ID":"0006c3a5-faa2-4b14-a57c-1e1b468041a8","Type":"ContainerStarted","Data":"7cd7b307523645bf5c0d079ef3c4abc919631bd982468580d94d9d3069f8e4ec"} Jul 29 08:21:27.995835 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.995816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-svvdf" event={"ID":"928e848d-cc8f-4abe-b018-ea9084a8f303","Type":"ContainerStarted","Data":"fbc25c0797564a0b733d2576fbef483c14d40b76042b92121dd586fbdabf1e67"} Jul 29 08:21:27.997190 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.997168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ssfvp" event={"ID":"c7e3136e-5a0b-4c82-a3b4-5d43b35d442e","Type":"ContainerStarted","Data":"bfd3ee3a5a3d1e2ac571ac8bdf95bdf6f4fc07007ff1a0637f065be1c04e51a6"} Jul 29 08:21:27.999011 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.998993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"a2e5585720e802660c9fac968cd27f4ec31fdf124fabe7b987d5b85046805dab"} Jul 29 08:21:27.999105 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:27.999061 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"14fd1c6b541db45b6c6b3d7c22e421bf98ec9a8069220afc2abda361f34887b6"} Jul 29 08:21:28.000348 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.000323 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ebeefc4-a87d-46db-8f40-d2cebb70eca1" containerID="7064678bb2f5c3fa36c84896ad5201fc7cf5a056cf8e93900b1834c9c336f903" exitCode=0 Jul 29 08:21:28.000448 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.000391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerDied","Data":"7064678bb2f5c3fa36c84896ad5201fc7cf5a056cf8e93900b1834c9c336f903"} Jul 29 08:21:28.002275 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.002257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" event={"ID":"40efe082-36d8-470a-9c42-754292673df6","Type":"ContainerStarted","Data":"2bba1af04bf25c149c7801169a0c43e98964c8177694a9216a430c2d6d77bb8e"} Jul 29 08:21:28.078262 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.078223 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m7c5r" podStartSLOduration=3.188366838 podStartE2EDuration="22.078210487s" totalImagesPullingTime="18.889843649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:28.078033228 +0000 UTC m=+22.657714911" watchObservedRunningTime="2026-07-29 08:21:28.078210487 +0000 UTC m=+22.657892167" Jul 29 08:21:28.078473 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.078453 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lw44r" podStartSLOduration=4.201135599 podStartE2EDuration="23.078448233s" totalImagesPullingTime="18.877312634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:28.043113089 +0000 UTC m=+22.622794772" watchObservedRunningTime="2026-07-29 08:21:28.078448233 +0000 UTC m=+22.658129906" Jul 29 08:21:28.133400 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.133356 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w5j4b" podStartSLOduration=3.277877007 podStartE2EDuration="22.13334228s" totalImagesPullingTime="18.855465273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:28.106915184 +0000 UTC m=+22.686596868" watchObservedRunningTime="2026-07-29 08:21:28.13334228 +0000 UTC m=+22.713023963" Jul 29 08:21:28.159911 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.159874 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-svvdf" podStartSLOduration=3.247769342 podStartE2EDuration="22.159859271s" totalImagesPullingTime="18.912089929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:28.133767205 +0000 UTC m=+22.713448888" watchObservedRunningTime="2026-07-29 08:21:28.159859271 +0000 UTC m=+22.739540954" Jul 29 08:21:28.226264 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.226214 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ssfvp" podStartSLOduration=3.37111091 podStartE2EDuration="22.226200148s" totalImagesPullingTime="18.855089238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:28.160557568 +0000 UTC m=+22.740239253" watchObservedRunningTime="2026-07-29 08:21:28.226200148 +0000 UTC m=+22.805881831" Jul 29 08:21:28.895268 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.895247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:28.895414 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:28.895365 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:28.895498 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.895423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:28.895580 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:28.895562 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:28.926825 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:28.926799 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 08:21:29.007859 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.007826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"d9436fa7b8c7c12cb66a8efe1c6cabe5e9e72325ea5e39748f1fad79313b1843"} Jul 29 08:21:29.008023 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.007897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"d0062754060de3547da798ff3d3cd19ee5806145ced31893a07a8ba4205ff139"} Jul 29 08:21:29.008023 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.007940 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"2976e3e3a175084b907991140d806252a4700d11f78a383120edf29f0d80dedd"} Jul 29 08:21:29.008023 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.007987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"f4317df959f121b0175447f0acc7d036c87b8f085bb7727c4b0b4774f5891fb3"} Jul 29 08:21:29.009427 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.009401 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pb88v" event={"ID":"13fd06f4-1b82-4049-b980-0ef9d1ae0742","Type":"ContainerStarted","Data":"55373ae283e61ada9782dffbcfed2de0596b94c6f057fe67cbbb8ee59aebbe87"} Jul 29 08:21:29.011369 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.011342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" event={"ID":"3be7fa5e-32eb-4151-9f4a-9b80f2420d72","Type":"ContainerStarted","Data":"38dd7f1176676a1e8768f0390795cb9aae419a300d655aef6c54e143208ba863"} Jul 29 08:21:29.829678 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.829503 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T08:21:28.9268216Z","UUID":"fe672235-1e33-47eb-aefd-3d08372e6873","Handler":null,"Name":"","Endpoint":""} Jul 29 08:21:29.831251 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.831226 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 08:21:29.831394 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.831273 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 08:21:29.895996 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:29.895966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:29.896163 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:29.896096 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:30.591297 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:30.591264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:30.592008 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:30.591988 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:30.609405 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:30.609364 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pb88v" podStartSLOduration=5.743292087 podStartE2EDuration="24.609348115s" totalImagesPullingTime="18.866056028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:29.028383098 +0000 UTC m=+23.608064781" watchObservedRunningTime="2026-07-29 08:21:30.609348115 +0000 UTC m=+25.189029801" Jul 29 08:21:30.895451 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:30.895412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:30.895648 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:30.895545 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:30.895648 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:30.895595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:30.895781 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:30.895717 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:31.019161 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:31.018190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" event={"ID":"3be7fa5e-32eb-4151-9f4a-9b80f2420d72","Type":"ContainerStarted","Data":"b79b11286c26c7fc841a1ebe4469cbc14759c21efc4b6bec7d53aa8d4457bb11"} Jul 29 08:21:31.024582 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:31.024555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"64af1074befe13859e74b5d53bbcba5c80002b03c8d125be4e3ad6eeba4f6ee0"} Jul 29 08:21:31.024822 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:31.024804 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:31.025321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:31.025305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ssfvp" Jul 29 08:21:31.040426 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:31.040387 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6mxr" podStartSLOduration=5.101390951 podStartE2EDuration="26.040374653s" totalImagesPullingTime="20.938983702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:31.040293586 +0000 UTC m=+25.619975272" watchObservedRunningTime="2026-07-29 08:21:31.040374653 +0000 UTC m=+25.620056336" Jul 29 08:21:31.895753 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:31.895547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:31.896224 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:31.895834 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:32.895370 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:32.895150 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:32.895547 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:32.895151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:32.895547 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:32.895393 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:32.895547 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:32.895502 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:33.029782 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.029750 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ebeefc4-a87d-46db-8f40-d2cebb70eca1" containerID="386928b7b85fd25c2558d5572757379232d4ac29cdc91cbe3d1e68d9b556daff" exitCode=0 Jul 29 08:21:33.030642 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.029837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerDied","Data":"386928b7b85fd25c2558d5572757379232d4ac29cdc91cbe3d1e68d9b556daff"} Jul 29 08:21:33.033503 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.033477 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" event={"ID":"222f904f-3dff-40ac-a735-40260879417f","Type":"ContainerStarted","Data":"77723300e2141a9524401b56e8566f5b0fa9c690a5eca95f4328ec5c6f1525e2"} Jul 29 08:21:33.033728 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.033698 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:33.033830 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.033796 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:33.050037 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.050011 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:33.085299 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.085245 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" podStartSLOduration=8.142107783 podStartE2EDuration="27.085228684s" totalImagesPullingTime="18.943120901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:33.084755072 +0000 UTC m=+27.664436754" watchObservedRunningTime="2026-07-29 08:21:33.085228684 +0000 UTC m=+27.664910366" Jul 29 08:21:33.897636 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:33.897602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:33.897941 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:33.897731 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:34.035762 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.035731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:34.051043 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.051008 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:21:34.564783 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.564753 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vmj9l"] Jul 29 08:21:34.564955 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.564925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:34.565070 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:34.565050 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:34.565599 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.565570 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fztml"] Jul 29 08:21:34.565690 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.565641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:34.565733 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:34.565706 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:34.566574 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.566551 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqzk8"] Jul 29 08:21:34.566680 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:34.566643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:34.566750 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:34.566733 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:35.039471 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:35.039418 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ebeefc4-a87d-46db-8f40-d2cebb70eca1" containerID="ff01a1c7beebcb15710895478a440650f8f4051b6c2a24417f5b9fa92cb634a8" exitCode=0 Jul 29 08:21:35.039936 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:35.039508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerDied","Data":"ff01a1c7beebcb15710895478a440650f8f4051b6c2a24417f5b9fa92cb634a8"} Jul 29 08:21:35.895818 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:35.895778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:35.896169 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:35.895891 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:35.896169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:35.895906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:35.896169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:35.895931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:35.896169 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:35.896030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:35.896169 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:35.896121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:37.045934 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:37.045793 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ebeefc4-a87d-46db-8f40-d2cebb70eca1" containerID="ab35a575afdadfc7d88ab171c47a569b50744826c02a957578bd7ad1669c8e41" exitCode=0 Jul 29 08:21:37.045934 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:37.045850 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerDied","Data":"ab35a575afdadfc7d88ab171c47a569b50744826c02a957578bd7ad1669c8e41"} Jul 29 08:21:37.895996 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:37.895959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:37.896184 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:37.895960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:37.896184 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:37.896102 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:37.896184 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:37.895974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:37.896184 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:37.896168 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:37.896345 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:37.896224 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:39.512230 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:39.512186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:39.512230 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:39.512236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:39.512918 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.512333 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:39.512918 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.512354 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:39.512918 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.512399 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret podName:62056413-e3c1-4562-b6ae-6ee8a49df104 nodeName:}" failed. No retries permitted until 2026-07-29 08:22:11.51238085 +0000 UTC m=+66.092062534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret") pod "global-pull-secret-syncer-fztml" (UID: "62056413-e3c1-4562-b6ae-6ee8a49df104") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:21:39.512918 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.512419 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:22:11.512408987 +0000 UTC m=+66.092090649 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:21:39.613453 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:39.613399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:39.613608 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.613583 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:21:39.613608 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.613606 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:21:39.613714 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.613619 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7v4fb for pod openshift-network-diagnostics/network-check-target-hqzk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:39.613714 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.613670 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb podName:a5014e06-bb11-4b83-854d-74f033f51cb9 nodeName:}" failed. No retries permitted until 2026-07-29 08:22:11.613656116 +0000 UTC m=+66.193337777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7v4fb" (UniqueName: "kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb") pod "network-check-target-hqzk8" (UID: "a5014e06-bb11-4b83-854d-74f033f51cb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:21:39.895803 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:39.895724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:39.895968 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.895851 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fztml" podUID="62056413-e3c1-4562-b6ae-6ee8a49df104" Jul 29 08:21:39.895968 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:39.895729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:39.895968 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.895928 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqzk8" podUID="a5014e06-bb11-4b83-854d-74f033f51cb9" Jul 29 08:21:39.895968 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:39.895724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:39.896152 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:39.895987 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vmj9l" podUID="bf109907-5eb4-404b-8454-e87a1e2c5849" Jul 29 08:21:40.204165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.204133 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-229.ec2.internal" event="NodeReady" Jul 29 08:21:40.204333 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.204277 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 08:21:40.241120 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.241076 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c4f67556c-nt4x9"] Jul 29 08:21:40.244414 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.244390 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.247214 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.247191 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rzlxr\"" Jul 29 08:21:40.247334 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.247227 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 08:21:40.247334 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.247199 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 08:21:40.247473 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.247191 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 08:21:40.253187 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.252755 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 08:21:40.262553 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.262504 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-khm2f"] Jul 29 08:21:40.265093 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.265074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c4f67556c-nt4x9"] Jul 29 08:21:40.265093 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.265108 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9j5xf"] Jul 29 08:21:40.265272 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.265222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.267508 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.267487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.267757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.267732 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 08:21:40.268077 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.268055 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wsvft\"" Jul 29 08:21:40.268173 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.268154 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 08:21:40.269649 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.269625 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 08:21:40.273880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.270500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 08:21:40.273880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.270537 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 08:21:40.273880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.270694 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xzvt9\"" Jul 29 08:21:40.276645 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.276628 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khm2f"] Jul 29 08:21:40.278262 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.278241 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9j5xf"] Jul 29 08:21:40.319256 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-installation-pull-secrets\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.319484 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-certificates\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.319484 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.319484 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ce430d-bc66-44d9-9331-6d4cb913a93f-ca-trust-extracted\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.319484 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-image-registry-private-configuration\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.319695 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-trusted-ca\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.319695 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-bound-sa-token\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.319695 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.319597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbnv\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-kube-api-access-vqbnv\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.420745 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-image-registry-private-configuration\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.420926 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b6c47ce-1287-4bef-80d5-42b69b8c5431-cert\") pod \"ingress-canary-9j5xf\" (UID: \"7b6c47ce-1287-4bef-80d5-42b69b8c5431\") " pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.420926 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrzsp\" (UniqueName: \"kubernetes.io/projected/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-kube-api-access-hrzsp\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.420926 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-trusted-ca\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.420926 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-bound-sa-token\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.421125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbnv\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-kube-api-access-vqbnv\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.421125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-installation-pull-secrets\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.421125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.420999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.421125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-certificates\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.421334 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.421385 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421356 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghlc\" (UniqueName: \"kubernetes.io/projected/7b6c47ce-1287-4bef-80d5-42b69b8c5431-kube-api-access-9ghlc\") pod \"ingress-canary-9j5xf\" (UID: \"7b6c47ce-1287-4bef-80d5-42b69b8c5431\") " pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.421466 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ce430d-bc66-44d9-9331-6d4cb913a93f-ca-trust-extracted\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.421466 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-config-volume\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.421466 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-tmp-dir\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.421625 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.421589 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:21:40.421625 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.421605 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c4f67556c-nt4x9: secret "image-registry-tls" not found Jul 29 08:21:40.421719 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.421683 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls podName:b3ce430d-bc66-44d9-9331-6d4cb913a93f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:40.921663474 +0000 UTC m=+35.501345153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls") pod "image-registry-c4f67556c-nt4x9" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f") : secret "image-registry-tls" not found Jul 29 08:21:40.421786 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-certificates\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.421786 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ce430d-bc66-44d9-9331-6d4cb913a93f-ca-trust-extracted\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.422189 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.421963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-trusted-ca\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.425788 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.425767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-image-registry-private-configuration\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.425897 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.425770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-installation-pull-secrets\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.430797 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.430778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-bound-sa-token\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.432100 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.431496 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbnv\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-kube-api-access-vqbnv\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.522736 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.522645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b6c47ce-1287-4bef-80d5-42b69b8c5431-cert\") pod \"ingress-canary-9j5xf\" (UID: \"7b6c47ce-1287-4bef-80d5-42b69b8c5431\") " pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.522736 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.522690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrzsp\" (UniqueName: \"kubernetes.io/projected/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-kube-api-access-hrzsp\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.523412 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.522752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.523412 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.522788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghlc\" (UniqueName: \"kubernetes.io/projected/7b6c47ce-1287-4bef-80d5-42b69b8c5431-kube-api-access-9ghlc\") pod \"ingress-canary-9j5xf\" (UID: \"7b6c47ce-1287-4bef-80d5-42b69b8c5431\") " pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.523412 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.522809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-config-volume\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.523412 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.522835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-tmp-dir\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.523412 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.522895 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:21:40.523412 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.522966 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls podName:5121a769-ab74-4c2e-8f1e-d7c09c88d44f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:41.022947301 +0000 UTC m=+35.602628966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls") pod "dns-default-khm2f" (UID: "5121a769-ab74-4c2e-8f1e-d7c09c88d44f") : secret "dns-default-metrics-tls" not found Jul 29 08:21:40.523412 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.523171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-tmp-dir\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.523717 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.523659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-config-volume\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.525420 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.525394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b6c47ce-1287-4bef-80d5-42b69b8c5431-cert\") pod \"ingress-canary-9j5xf\" (UID: \"7b6c47ce-1287-4bef-80d5-42b69b8c5431\") " pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.532202 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.532180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrzsp\" (UniqueName: \"kubernetes.io/projected/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-kube-api-access-hrzsp\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:40.532310 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.532202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghlc\" (UniqueName: \"kubernetes.io/projected/7b6c47ce-1287-4bef-80d5-42b69b8c5431-kube-api-access-9ghlc\") pod \"ingress-canary-9j5xf\" (UID: \"7b6c47ce-1287-4bef-80d5-42b69b8c5431\") " pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.587528 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.587489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9j5xf" Jul 29 08:21:40.926865 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:40.926819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:40.927033 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.926976 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:21:40.927033 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.926996 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c4f67556c-nt4x9: secret "image-registry-tls" not found Jul 29 08:21:40.927139 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:40.927063 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls podName:b3ce430d-bc66-44d9-9331-6d4cb913a93f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:41.927046785 +0000 UTC m=+36.506728450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls") pod "image-registry-c4f67556c-nt4x9" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f") : secret "image-registry-tls" not found Jul 29 08:21:41.028041 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.028004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:41.028208 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:41.028146 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:21:41.028284 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:41.028214 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls podName:5121a769-ab74-4c2e-8f1e-d7c09c88d44f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:42.028195348 +0000 UTC m=+36.607877024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls") pod "dns-default-khm2f" (UID: "5121a769-ab74-4c2e-8f1e-d7c09c88d44f") : secret "dns-default-metrics-tls" not found Jul 29 08:21:41.289049 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.288894 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9j5xf"] Jul 29 08:21:41.292568 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:21:41.292546 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6c47ce_1287_4bef_80d5_42b69b8c5431.slice/crio-e1e611e1a74536fdfb0f748b833887f6087183132cc4761a863ae163a9d58d03 WatchSource:0}: Error finding container e1e611e1a74536fdfb0f748b833887f6087183132cc4761a863ae163a9d58d03: Status 404 returned error can't find the container with id e1e611e1a74536fdfb0f748b833887f6087183132cc4761a863ae163a9d58d03 Jul 29 08:21:41.896068 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.896007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:21:41.896068 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.896032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:21:41.896598 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.896007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:21:41.899037 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.899008 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 08:21:41.899155 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.899041 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 08:21:41.899155 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.899020 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 08:21:41.899923 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.899893 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 08:21:41.900041 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.899983 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7c8hq\"" Jul 29 08:21:41.900041 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.900001 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wscl8\"" Jul 29 08:21:41.936658 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:41.936628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:41.936834 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:41.936801 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:21:41.936834 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:41.936817 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c4f67556c-nt4x9: secret "image-registry-tls" not found Jul 29 08:21:41.936931 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:41.936875 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls podName:b3ce430d-bc66-44d9-9331-6d4cb913a93f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:43.93685713 +0000 UTC m=+38.516538792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls") pod "image-registry-c4f67556c-nt4x9" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f") : secret "image-registry-tls" not found Jul 29 08:21:42.037507 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:42.037461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:42.037673 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:42.037642 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:21:42.037729 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:42.037722 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls podName:5121a769-ab74-4c2e-8f1e-d7c09c88d44f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:44.037701717 +0000 UTC m=+38.617383384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls") pod "dns-default-khm2f" (UID: "5121a769-ab74-4c2e-8f1e-d7c09c88d44f") : secret "dns-default-metrics-tls" not found Jul 29 08:21:42.060509 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:42.060466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9j5xf" event={"ID":"7b6c47ce-1287-4bef-80d5-42b69b8c5431","Type":"ContainerStarted","Data":"e1e611e1a74536fdfb0f748b833887f6087183132cc4761a863ae163a9d58d03"} Jul 29 08:21:42.063500 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:42.063472 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ebeefc4-a87d-46db-8f40-d2cebb70eca1" containerID="c821266d80b613a5139b4898ac75b4ad39187a48abceec71e881317b183d17d0" exitCode=0 Jul 29 08:21:42.063617 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:42.063506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerDied","Data":"c821266d80b613a5139b4898ac75b4ad39187a48abceec71e881317b183d17d0"} Jul 29 08:21:43.069097 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:43.069060 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ebeefc4-a87d-46db-8f40-d2cebb70eca1" containerID="077e19e0eccef28e436fab13f1647bdd62acdde710b840d06213a2304641139f" exitCode=0 Jul 29 08:21:43.069591 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:43.069148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerDied","Data":"077e19e0eccef28e436fab13f1647bdd62acdde710b840d06213a2304641139f"} Jul 29 08:21:43.953468 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:43.953286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:43.953609 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:43.953454 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:21:43.953609 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:43.953541 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c4f67556c-nt4x9: secret "image-registry-tls" not found Jul 29 08:21:43.953609 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:43.953596 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls podName:b3ce430d-bc66-44d9-9331-6d4cb913a93f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:47.953580705 +0000 UTC m=+42.533262366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls") pod "image-registry-c4f67556c-nt4x9" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f") : secret "image-registry-tls" not found Jul 29 08:21:44.054694 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:44.054665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:44.054869 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:44.054848 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:21:44.054936 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:44.054925 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls podName:5121a769-ab74-4c2e-8f1e-d7c09c88d44f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:48.054905713 +0000 UTC m=+42.634587376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls") pod "dns-default-khm2f" (UID: "5121a769-ab74-4c2e-8f1e-d7c09c88d44f") : secret "dns-default-metrics-tls" not found Jul 29 08:21:44.074489 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:44.074455 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" event={"ID":"5ebeefc4-a87d-46db-8f40-d2cebb70eca1","Type":"ContainerStarted","Data":"0a280a4837085d9a2c6e44f55b3063d60b6c14f75d1045a5baeb223e5a9df229"} Jul 29 08:21:44.075891 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:44.075870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9j5xf" event={"ID":"7b6c47ce-1287-4bef-80d5-42b69b8c5431","Type":"ContainerStarted","Data":"00444c8df88ffd278c7c4c8e29efe1cc66cf006f881fa6f2944fe45984a123b0"} Jul 29 08:21:44.098731 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:44.098677 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p2w7k" podStartSLOduration=8.69864563 podStartE2EDuration="38.098661349s" totalImagesPullingTime="29.400015719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:44.097873206 +0000 UTC m=+38.677554902" watchObservedRunningTime="2026-07-29 08:21:44.098661349 +0000 UTC m=+38.678343009" Jul 29 08:21:44.115870 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:44.115802 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9j5xf" podStartSLOduration=1.542752343 podStartE2EDuration="4.115779758s" totalImagesPullingTime="2.573027415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:21:44.114617748 +0000 UTC m=+38.694299432" watchObservedRunningTime="2026-07-29 08:21:44.115779758 +0000 UTC m=+38.695461442" Jul 29 08:21:47.986698 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:47.986662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:47.987182 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:47.986799 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:21:47.987182 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:47.986813 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c4f67556c-nt4x9: secret "image-registry-tls" not found Jul 29 08:21:47.987182 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:47.986879 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls podName:b3ce430d-bc66-44d9-9331-6d4cb913a93f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:55.986858657 +0000 UTC m=+50.566540318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls") pod "image-registry-c4f67556c-nt4x9" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f") : secret "image-registry-tls" not found Jul 29 08:21:48.087692 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:48.087667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:48.087821 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:48.087796 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:21:48.087858 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:48.087849 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls podName:5121a769-ab74-4c2e-8f1e-d7c09c88d44f nodeName:}" failed. No retries permitted until 2026-07-29 08:21:56.087834382 +0000 UTC m=+50.667516046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls") pod "dns-default-khm2f" (UID: "5121a769-ab74-4c2e-8f1e-d7c09c88d44f") : secret "dns-default-metrics-tls" not found Jul 29 08:21:56.046448 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:56.046394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:21:56.046883 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:56.046517 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:21:56.046883 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:56.046528 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c4f67556c-nt4x9: secret "image-registry-tls" not found Jul 29 08:21:56.046883 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:56.046575 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls podName:b3ce430d-bc66-44d9-9331-6d4cb913a93f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:12.046561061 +0000 UTC m=+66.626242722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls") pod "image-registry-c4f67556c-nt4x9" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f") : secret "image-registry-tls" not found Jul 29 08:21:56.146684 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:21:56.146652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:21:56.146834 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:56.146791 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:21:56.146883 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:21:56.146854 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls podName:5121a769-ab74-4c2e-8f1e-d7c09c88d44f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:12.146839149 +0000 UTC m=+66.726520811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls") pod "dns-default-khm2f" (UID: "5121a769-ab74-4c2e-8f1e-d7c09c88d44f") : secret "dns-default-metrics-tls" not found Jul 29 08:22:06.061200 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:06.061176 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ffbdj" Jul 29 08:22:08.093397 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.093367 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb"] Jul 29 08:22:08.095804 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.095783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.099453 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.099398 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:22:08.099618 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.099467 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 08:22:08.099690 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.099612 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 08:22:08.099690 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.099670 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 08:22:08.099791 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.099734 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wlrwc\"" Jul 29 08:22:08.107355 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.107333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb"] Jul 29 08:22:08.123927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.123905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.124017 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.123965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqjq\" (UniqueName: \"kubernetes.io/projected/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-kube-api-access-9gqjq\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.124066 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.124025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-config\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.224843 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.224824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqjq\" (UniqueName: \"kubernetes.io/projected/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-kube-api-access-9gqjq\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.224944 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.224859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-config\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.224944 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.224887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.225373 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.225345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-config\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.228594 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.228575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.234563 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.234536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqjq\" (UniqueName: \"kubernetes.io/projected/00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a-kube-api-access-9gqjq\") pod \"service-ca-operator-5d6c75f84f-fcsxb\" (UID: \"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.408010 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.407991 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" Jul 29 08:22:08.525678 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:08.525643 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb"] Jul 29 08:22:08.528418 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:08.528391 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d457a6_d3fe_4cf0_92e0_66c6e92bbe9a.slice/crio-413f903b176e9e4a14ef84be33dd9ce35aab4b0f0c0cde0b818c10e2cfd514b2 WatchSource:0}: Error finding container 413f903b176e9e4a14ef84be33dd9ce35aab4b0f0c0cde0b818c10e2cfd514b2: Status 404 returned error can't find the container with id 413f903b176e9e4a14ef84be33dd9ce35aab4b0f0c0cde0b818c10e2cfd514b2 Jul 29 08:22:09.135448 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:09.135389 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" event={"ID":"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a","Type":"ContainerStarted","Data":"413f903b176e9e4a14ef84be33dd9ce35aab4b0f0c0cde0b818c10e2cfd514b2"} Jul 29 08:22:11.141208 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.141174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" event={"ID":"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a","Type":"ContainerStarted","Data":"e7dfe04033cfc61cafa9b7341b3fbb691b6c770f3534714075f51edfe2758a79"} Jul 29 08:22:11.159811 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.159762 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" podStartSLOduration=1.396235841 podStartE2EDuration="3.159745569s" totalImagesPullingTime="1.763509728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:11.159199041 +0000 UTC m=+65.738880725" watchObservedRunningTime="2026-07-29 08:22:11.159745569 +0000 UTC m=+65.739427256" Jul 29 08:22:11.550639 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.550610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:22:11.550843 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.550684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:22:11.553497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.553475 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 08:22:11.553605 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.553588 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 08:22:11.561543 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:11.561521 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 08:22:11.561612 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:11.561579 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs podName:bf109907-5eb4-404b-8454-e87a1e2c5849 nodeName:}" failed. No retries permitted until 2026-07-29 08:23:15.561560309 +0000 UTC m=+130.141241969 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs") pod "network-metrics-daemon-vmj9l" (UID: "bf109907-5eb4-404b-8454-e87a1e2c5849") : secret "metrics-daemon-secret" not found Jul 29 08:22:11.563670 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.563652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62056413-e3c1-4562-b6ae-6ee8a49df104-original-pull-secret\") pod \"global-pull-secret-syncer-fztml\" (UID: \"62056413-e3c1-4562-b6ae-6ee8a49df104\") " pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:22:11.615937 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.615904 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fztml" Jul 29 08:22:11.651780 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.651344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:22:11.654425 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.654405 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 08:22:11.665262 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.665076 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 08:22:11.675567 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.675512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4fb\" (UniqueName: \"kubernetes.io/projected/a5014e06-bb11-4b83-854d-74f033f51cb9-kube-api-access-7v4fb\") pod \"network-check-target-hqzk8\" (UID: \"a5014e06-bb11-4b83-854d-74f033f51cb9\") " pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:22:11.743163 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.743132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fztml"] Jul 29 08:22:11.746453 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:11.746407 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62056413_e3c1_4562_b6ae_6ee8a49df104.slice/crio-67091e66064bffbe64843b43cdc9978487baafcc2a79caf7b063ff443ad78819 WatchSource:0}: Error finding container 67091e66064bffbe64843b43cdc9978487baafcc2a79caf7b063ff443ad78819: Status 404 returned error can't find the container with id 67091e66064bffbe64843b43cdc9978487baafcc2a79caf7b063ff443ad78819 Jul 29 08:22:11.911663 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.911643 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wscl8\"" Jul 29 08:22:11.919961 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:11.919944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:22:12.043688 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:12.043665 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqzk8"] Jul 29 08:22:12.047535 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:12.047515 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5014e06_bb11_4b83_854d_74f033f51cb9.slice/crio-651f7f6213381530d40a1e078f89921be98c8d59179a6e240b7679810f3ad529 WatchSource:0}: Error finding container 651f7f6213381530d40a1e078f89921be98c8d59179a6e240b7679810f3ad529: Status 404 returned error can't find the container with id 651f7f6213381530d40a1e078f89921be98c8d59179a6e240b7679810f3ad529 Jul 29 08:22:12.054310 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:12.054247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:22:12.054416 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:12.054341 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:22:12.054416 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:12.054351 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c4f67556c-nt4x9: secret "image-registry-tls" not found Jul 29 08:22:12.054416 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:12.054399 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls podName:b3ce430d-bc66-44d9-9331-6d4cb913a93f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:44.05438178 +0000 UTC m=+98.634063443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls") pod "image-registry-c4f67556c-nt4x9" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f") : secret "image-registry-tls" not found Jul 29 08:22:12.144645 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:12.144607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqzk8" event={"ID":"a5014e06-bb11-4b83-854d-74f033f51cb9","Type":"ContainerStarted","Data":"651f7f6213381530d40a1e078f89921be98c8d59179a6e240b7679810f3ad529"} Jul 29 08:22:12.145742 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:12.145716 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fztml" event={"ID":"62056413-e3c1-4562-b6ae-6ee8a49df104","Type":"ContainerStarted","Data":"67091e66064bffbe64843b43cdc9978487baafcc2a79caf7b063ff443ad78819"} Jul 29 08:22:12.155177 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:12.155158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:22:12.155316 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:12.155301 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:22:12.155368 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:12.155359 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls podName:5121a769-ab74-4c2e-8f1e-d7c09c88d44f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:44.155342053 +0000 UTC m=+98.735023716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls") pod "dns-default-khm2f" (UID: "5121a769-ab74-4c2e-8f1e-d7c09c88d44f") : secret "dns-default-metrics-tls" not found Jul 29 08:22:12.997535 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:12.997381 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5zbsm"] Jul 29 08:22:13.010554 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.010510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.013517 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.013263 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 08:22:13.014772 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.014703 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5zbsm"] Jul 29 08:22:13.015402 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.014986 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 08:22:13.015402 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.015057 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 08:22:13.015402 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.014997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 08:22:13.015876 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.015849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x95nv\"" Jul 29 08:22:13.063226 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.063193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.063345 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.063255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff71c6ab-1aa9-4620-a002-c09e43bd308f-data-volume\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.063345 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.063288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.063345 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.063315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkpd\" (UniqueName: \"kubernetes.io/projected/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-api-access-ffkpd\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.063494 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.063381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff71c6ab-1aa9-4620-a002-c09e43bd308f-crio-socket\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.164534 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.164483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff71c6ab-1aa9-4620-a002-c09e43bd308f-crio-socket\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.164930 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.164559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.164930 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.164593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff71c6ab-1aa9-4620-a002-c09e43bd308f-data-volume\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.164930 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.164628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.164930 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.164656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkpd\" (UniqueName: \"kubernetes.io/projected/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-api-access-ffkpd\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.164930 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.164671 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:22:13.164930 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.164761 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:13.664738934 +0000 UTC m=+68.244420606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : secret "insights-runtime-extractor-tls" not found Jul 29 08:22:13.164930 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.164588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff71c6ab-1aa9-4620-a002-c09e43bd308f-crio-socket\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.165289 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.164989 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:22:13.165289 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.165047 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:13.665031132 +0000 UTC m=+68.244712827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : configmap "kube-rbac-proxy" not found Jul 29 08:22:13.165289 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.165191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff71c6ab-1aa9-4620-a002-c09e43bd308f-data-volume\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.174995 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.174759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkpd\" (UniqueName: \"kubernetes.io/projected/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-api-access-ffkpd\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.670118 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.670080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.670292 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:13.670151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:13.670292 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.670262 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:22:13.670411 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.670327 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:14.670306157 +0000 UTC m=+69.249987819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : secret "insights-runtime-extractor-tls" not found Jul 29 08:22:13.670639 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.670262 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:22:13.670639 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:13.670543 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:14.67052213 +0000 UTC m=+69.250203814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : configmap "kube-rbac-proxy" not found Jul 29 08:22:14.320659 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.320628 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m2rgk"] Jul 29 08:22:14.346856 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.346799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m2rgk"] Jul 29 08:22:14.347000 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.346904 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.349978 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.349776 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 29 08:22:14.349978 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.349775 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 29 08:22:14.349978 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.349930 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 29 08:22:14.350831 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.350798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 29 08:22:14.351023 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.351006 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kzqdl\"" Jul 29 08:22:14.375081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.375056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78244da5-7fdd-4da8-920f-67d56984eaee-signing-key\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.375207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.375151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78244da5-7fdd-4da8-920f-67d56984eaee-signing-cabundle\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.375207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.375182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6xr\" (UniqueName: \"kubernetes.io/projected/78244da5-7fdd-4da8-920f-67d56984eaee-kube-api-access-5m6xr\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.476132 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.476100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78244da5-7fdd-4da8-920f-67d56984eaee-signing-key\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.476282 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.476167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78244da5-7fdd-4da8-920f-67d56984eaee-signing-cabundle\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.476282 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.476213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6xr\" (UniqueName: \"kubernetes.io/projected/78244da5-7fdd-4da8-920f-67d56984eaee-kube-api-access-5m6xr\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.477011 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.476987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78244da5-7fdd-4da8-920f-67d56984eaee-signing-cabundle\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.479262 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.479242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78244da5-7fdd-4da8-920f-67d56984eaee-signing-key\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.486257 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.486232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6xr\" (UniqueName: \"kubernetes.io/projected/78244da5-7fdd-4da8-920f-67d56984eaee-kube-api-access-5m6xr\") pod \"service-ca-658d7b7c78-m2rgk\" (UID: \"78244da5-7fdd-4da8-920f-67d56984eaee\") " pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.658847 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.658812 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" Jul 29 08:22:14.678725 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.678701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:14.678832 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:14.678786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:14.678885 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:14.678831 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:22:14.678935 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:14.678890 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:22:14.678935 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:14.678900 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:16.678881482 +0000 UTC m=+71.258563157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : configmap "kube-rbac-proxy" not found Jul 29 08:22:14.678935 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:14.678924 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:16.678914049 +0000 UTC m=+71.258595709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : secret "insights-runtime-extractor-tls" not found Jul 29 08:22:15.792675 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:15.792649 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lw44r_5beacbc7-a61e-4621-912e-d3807d189d2d/dns-node-resolver/0.log" Jul 29 08:22:16.698123 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:16.698101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:16.698285 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:16.698182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:16.698285 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:16.698266 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:22:16.698368 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:16.698309 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:22:16.698368 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:16.698337 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:20.698317542 +0000 UTC m=+75.277999213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : secret "insights-runtime-extractor-tls" not found Jul 29 08:22:16.698485 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:16.698377 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:20.698359769 +0000 UTC m=+75.278041453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : configmap "kube-rbac-proxy" not found Jul 29 08:22:16.755487 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:16.755461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m2rgk"] Jul 29 08:22:16.758012 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:16.757989 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78244da5_7fdd_4da8_920f_67d56984eaee.slice/crio-dbaf85ac48f6aca0a7c5da2f1a2da5f0f60e10e4da70a8cb70aadfb156b0c591 WatchSource:0}: Error finding container dbaf85ac48f6aca0a7c5da2f1a2da5f0f60e10e4da70a8cb70aadfb156b0c591: Status 404 returned error can't find the container with id dbaf85ac48f6aca0a7c5da2f1a2da5f0f60e10e4da70a8cb70aadfb156b0c591 Jul 29 08:22:16.994379 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:16.994350 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w5j4b_0006c3a5-faa2-4b14-a57c-1e1b468041a8/node-ca/0.log" Jul 29 08:22:17.160731 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.160697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqzk8" event={"ID":"a5014e06-bb11-4b83-854d-74f033f51cb9","Type":"ContainerStarted","Data":"83d0196fcabed0a09a8048345b179f562695a6b7b81031575cb9a4f07aa0b372"} Jul 29 08:22:17.160945 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.160839 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:22:17.162368 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.162337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" event={"ID":"78244da5-7fdd-4da8-920f-67d56984eaee","Type":"ContainerStarted","Data":"f0dc8e8a313f64cf3962426a34fc686dda4f66f801f5bf101ed9390339d674bc"} Jul 29 08:22:17.162523 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.162378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" event={"ID":"78244da5-7fdd-4da8-920f-67d56984eaee","Type":"ContainerStarted","Data":"dbaf85ac48f6aca0a7c5da2f1a2da5f0f60e10e4da70a8cb70aadfb156b0c591"} Jul 29 08:22:17.163885 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.163857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fztml" event={"ID":"62056413-e3c1-4562-b6ae-6ee8a49df104","Type":"ContainerStarted","Data":"e09cd388edff56ec14979a667e5dbb8d30deec291136c76212cecbe4f451fa23"} Jul 29 08:22:17.178861 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.178816 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hqzk8" podStartSLOduration=66.597902929 podStartE2EDuration="1m11.178804143s" totalImagesPullingTime="4.580901214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:17.17781079 +0000 UTC m=+71.757492485" watchObservedRunningTime="2026-07-29 08:22:17.178804143 +0000 UTC m=+71.758485803" Jul 29 08:22:17.196851 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.196806 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fztml" podStartSLOduration=66.309159903 podStartE2EDuration="1m11.19678963s" totalImagesPullingTime="4.887629727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:17.196042744 +0000 UTC m=+71.775724427" watchObservedRunningTime="2026-07-29 08:22:17.19678963 +0000 UTC m=+71.776471313" Jul 29 08:22:17.213733 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.213694 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-m2rgk" podStartSLOduration=3.213684107 podStartE2EDuration="3.213684107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:17.213547368 +0000 UTC m=+71.793229052" watchObservedRunningTime="2026-07-29 08:22:17.213684107 +0000 UTC m=+71.793365789" Jul 29 08:22:17.591532 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:17.591504 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9j5xf_7b6c47ce-1287-4bef-80d5-42b69b8c5431/serve-healthcheck-canary/0.log" Jul 29 08:22:20.727780 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:20.727744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:20.728179 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:20.727816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:20.728179 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:20.727866 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:22:20.728179 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:20.727911 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:22:20.728179 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:20.727928 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:28.72791314 +0000 UTC m=+83.307594802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : configmap "kube-rbac-proxy" not found Jul 29 08:22:20.728179 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:20.727949 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:28.727938286 +0000 UTC m=+83.307619947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : secret "insights-runtime-extractor-tls" not found Jul 29 08:22:28.785322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:28.785196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:28.785322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:28.785269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:28.785867 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:28.785355 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:22:28.785867 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:22:28.785423 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls podName:ff71c6ab-1aa9-4620-a002-c09e43bd308f nodeName:}" failed. No retries permitted until 2026-07-29 08:22:44.78540749 +0000 UTC m=+99.365089151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zbsm" (UID: "ff71c6ab-1aa9-4620-a002-c09e43bd308f") : secret "insights-runtime-extractor-tls" not found Jul 29 08:22:28.785988 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:28.785937 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff71c6ab-1aa9-4620-a002-c09e43bd308f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:33.770112 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.770080 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t"] Jul 29 08:22:33.774259 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.774241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:33.777003 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.776985 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 08:22:33.778016 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.777998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 08:22:33.778106 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.778078 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-swx9r\"" Jul 29 08:22:33.780072 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.780053 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs"] Jul 29 08:22:33.783045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.783028 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:33.785230 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.785211 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 08:22:33.785325 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.785259 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 08:22:33.785519 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.785500 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 08:22:33.785722 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.785709 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 08:22:33.785858 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.785843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 08:22:33.786421 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.786407 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 08:22:33.786520 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.786473 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 08:22:33.793598 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.793579 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t"] Jul 29 08:22:33.798503 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.798483 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs"] Jul 29 08:22:33.821624 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.821594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18084e73-8406-4101-ac8e-7cb9325be37a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gwk8t\" (UID: \"18084e73-8406-4101-ac8e-7cb9325be37a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:33.821706 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.821630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18084e73-8406-4101-ac8e-7cb9325be37a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-gwk8t\" (UID: \"18084e73-8406-4101-ac8e-7cb9325be37a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:33.922213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-hub\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:33.922213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:33.922399 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:33.922399 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-ca\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:33.922399 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqntj\" (UniqueName: \"kubernetes.io/projected/2a4614be-c416-4c97-b612-a555c4d080b3-kube-api-access-tqntj\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:33.922513 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18084e73-8406-4101-ac8e-7cb9325be37a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gwk8t\" (UID: \"18084e73-8406-4101-ac8e-7cb9325be37a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:33.922513 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18084e73-8406-4101-ac8e-7cb9325be37a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-gwk8t\" (UID: \"18084e73-8406-4101-ac8e-7cb9325be37a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:33.922513 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.922491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4614be-c416-4c97-b612-a555c4d080b3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:33.923118 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.923102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18084e73-8406-4101-ac8e-7cb9325be37a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-gwk8t\" (UID: \"18084e73-8406-4101-ac8e-7cb9325be37a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:33.925059 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:33.925035 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/18084e73-8406-4101-ac8e-7cb9325be37a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gwk8t\" (UID: \"18084e73-8406-4101-ac8e-7cb9325be37a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:34.023281 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.023214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4614be-c416-4c97-b612-a555c4d080b3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.023281 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.023256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-hub\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.023281 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.023272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.023523 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.023380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.023523 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.023416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-ca\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.023523 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.023452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqntj\" (UniqueName: \"kubernetes.io/projected/2a4614be-c416-4c97-b612-a555c4d080b3-kube-api-access-tqntj\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.023952 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.023929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a4614be-c416-4c97-b612-a555c4d080b3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.025872 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.025845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.025985 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.025877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-hub\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.026197 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.026179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-ca\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.026248 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.026224 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a4614be-c416-4c97-b612-a555c4d080b3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.035187 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.035167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqntj\" (UniqueName: \"kubernetes.io/projected/2a4614be-c416-4c97-b612-a555c4d080b3-kube-api-access-tqntj\") pod \"cluster-proxy-proxy-agent-69b9b44f8-f74fs\" (UID: \"2a4614be-c416-4c97-b612-a555c4d080b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.083205 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.083177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" Jul 29 08:22:34.105122 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.105099 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" Jul 29 08:22:34.222578 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.222548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t"] Jul 29 08:22:34.226692 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:34.226664 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18084e73_8406_4101_ac8e_7cb9325be37a.slice/crio-0b0c2a1a017cab5b12b2dfb90c6952544fb3db5013e0795d8101b88cc450e667 WatchSource:0}: Error finding container 0b0c2a1a017cab5b12b2dfb90c6952544fb3db5013e0795d8101b88cc450e667: Status 404 returned error can't find the container with id 0b0c2a1a017cab5b12b2dfb90c6952544fb3db5013e0795d8101b88cc450e667 Jul 29 08:22:34.243174 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:34.243151 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs"] Jul 29 08:22:34.246181 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:34.246158 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a4614be_c416_4c97_b612_a555c4d080b3.slice/crio-7e887c8ebaed5da2191f73bb25a6a9c1c676c322bdf2085568be8c612ea10629 WatchSource:0}: Error finding container 7e887c8ebaed5da2191f73bb25a6a9c1c676c322bdf2085568be8c612ea10629: Status 404 returned error can't find the container with id 7e887c8ebaed5da2191f73bb25a6a9c1c676c322bdf2085568be8c612ea10629 Jul 29 08:22:35.220293 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:35.220250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" event={"ID":"2a4614be-c416-4c97-b612-a555c4d080b3","Type":"ContainerStarted","Data":"7e887c8ebaed5da2191f73bb25a6a9c1c676c322bdf2085568be8c612ea10629"} Jul 29 08:22:35.221875 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:35.221766 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" event={"ID":"18084e73-8406-4101-ac8e-7cb9325be37a","Type":"ContainerStarted","Data":"0b0c2a1a017cab5b12b2dfb90c6952544fb3db5013e0795d8101b88cc450e667"} Jul 29 08:22:36.227141 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:36.227096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" event={"ID":"18084e73-8406-4101-ac8e-7cb9325be37a","Type":"ContainerStarted","Data":"b2eae40e871d5b42da197627fe20d2276998067bcdf0cfd6e459b45392a4af3a"} Jul 29 08:22:36.246400 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:36.246348 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gwk8t" podStartSLOduration=1.952563229 podStartE2EDuration="3.246334219s" totalImagesPullingTime="1.29377099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:36.244581218 +0000 UTC m=+90.824262899" watchObservedRunningTime="2026-07-29 08:22:36.246334219 +0000 UTC m=+90.826015901" Jul 29 08:22:38.236396 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:38.236315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" event={"ID":"2a4614be-c416-4c97-b612-a555c4d080b3","Type":"ContainerStarted","Data":"220b5472e9523b005ecf7237deec92f60a94886b4b8d0ea71a102f0b63216478"} Jul 29 08:22:38.236396 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:38.236360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" event={"ID":"2a4614be-c416-4c97-b612-a555c4d080b3","Type":"ContainerStarted","Data":"fae50146fbd8e5deb8e7213808368fdcfab8235fb3ac3e38bb5b1fd80badeacb"} Jul 29 08:22:38.236396 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:38.236371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" event={"ID":"2a4614be-c416-4c97-b612-a555c4d080b3","Type":"ContainerStarted","Data":"2aaf5ef833639a89f60b8a124e064a68de673aa8746ff27ec29fdb08f759878e"} Jul 29 08:22:38.259682 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:38.259641 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69b9b44f8-f74fs" podStartSLOduration=1.538267131 podStartE2EDuration="5.259627562s" totalImagesPullingTime="3.721360431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:38.25832038 +0000 UTC m=+92.838002063" watchObservedRunningTime="2026-07-29 08:22:38.259627562 +0000 UTC m=+92.839309244" Jul 29 08:22:44.115956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.115922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:22:44.118283 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.118262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"image-registry-c4f67556c-nt4x9\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:22:44.158786 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.158761 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rzlxr\"" Jul 29 08:22:44.166516 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.166499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:22:44.217225 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.217189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:22:44.220151 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.220104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5121a769-ab74-4c2e-8f1e-d7c09c88d44f-metrics-tls\") pod \"dns-default-khm2f\" (UID: \"5121a769-ab74-4c2e-8f1e-d7c09c88d44f\") " pod="openshift-dns/dns-default-khm2f" Jul 29 08:22:44.303179 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.303157 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c4f67556c-nt4x9"] Jul 29 08:22:44.305543 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:44.305514 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ce430d_bc66_44d9_9331_6d4cb913a93f.slice/crio-a35f46acaa7ce4c0da36ff46193b9ea69f6299161c6090abe80f652fe49a69a5 WatchSource:0}: Error finding container a35f46acaa7ce4c0da36ff46193b9ea69f6299161c6090abe80f652fe49a69a5: Status 404 returned error can't find the container with id a35f46acaa7ce4c0da36ff46193b9ea69f6299161c6090abe80f652fe49a69a5 Jul 29 08:22:44.484990 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.484965 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wsvft\"" Jul 29 08:22:44.492548 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.492532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khm2f" Jul 29 08:22:44.616243 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.616210 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khm2f"] Jul 29 08:22:44.618839 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:44.618796 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5121a769_ab74_4c2e_8f1e_d7c09c88d44f.slice/crio-cca03b1f063d73c537295b9b8b29555d8516e58965241aecda6bffc14a963942 WatchSource:0}: Error finding container cca03b1f063d73c537295b9b8b29555d8516e58965241aecda6bffc14a963942: Status 404 returned error can't find the container with id cca03b1f063d73c537295b9b8b29555d8516e58965241aecda6bffc14a963942 Jul 29 08:22:44.821602 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.821519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:44.823885 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:44.823858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff71c6ab-1aa9-4620-a002-c09e43bd308f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zbsm\" (UID: \"ff71c6ab-1aa9-4620-a002-c09e43bd308f\") " pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:45.123174 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:45.123087 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5zbsm" Jul 29 08:22:45.262050 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:45.261976 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khm2f" event={"ID":"5121a769-ab74-4c2e-8f1e-d7c09c88d44f","Type":"ContainerStarted","Data":"cca03b1f063d73c537295b9b8b29555d8516e58965241aecda6bffc14a963942"} Jul 29 08:22:45.264579 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:45.264546 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" event={"ID":"b3ce430d-bc66-44d9-9331-6d4cb913a93f","Type":"ContainerStarted","Data":"c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47"} Jul 29 08:22:45.264713 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:45.264598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" event={"ID":"b3ce430d-bc66-44d9-9331-6d4cb913a93f","Type":"ContainerStarted","Data":"a35f46acaa7ce4c0da36ff46193b9ea69f6299161c6090abe80f652fe49a69a5"} Jul 29 08:22:45.264713 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:45.264637 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:22:45.265106 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:45.265082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5zbsm"] Jul 29 08:22:45.268815 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:22:45.268779 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff71c6ab_1aa9_4620_a002_c09e43bd308f.slice/crio-2d15ffd16fd0b893ea7b21aefde32eb0aef060ce116314371f6fc90e7d21d838 WatchSource:0}: Error finding container 2d15ffd16fd0b893ea7b21aefde32eb0aef060ce116314371f6fc90e7d21d838: Status 404 returned error can't find the container with id 2d15ffd16fd0b893ea7b21aefde32eb0aef060ce116314371f6fc90e7d21d838 Jul 29 08:22:45.288814 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:45.288762 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" podStartSLOduration=99.288746232 podStartE2EDuration="1m39.288746232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:45.288500858 +0000 UTC m=+99.868182542" watchObservedRunningTime="2026-07-29 08:22:45.288746232 +0000 UTC m=+99.868427915" Jul 29 08:22:46.271538 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:46.271504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zbsm" event={"ID":"ff71c6ab-1aa9-4620-a002-c09e43bd308f","Type":"ContainerStarted","Data":"2d15ffd16fd0b893ea7b21aefde32eb0aef060ce116314371f6fc90e7d21d838"} Jul 29 08:22:47.277105 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:47.277039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khm2f" event={"ID":"5121a769-ab74-4c2e-8f1e-d7c09c88d44f","Type":"ContainerStarted","Data":"cf6e304ba15ca567fd0cc45dbe500f6967f1b15a91b061fcce11a2f69ddf3de8"} Jul 29 08:22:47.277593 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:47.277162 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khm2f" event={"ID":"5121a769-ab74-4c2e-8f1e-d7c09c88d44f","Type":"ContainerStarted","Data":"d3009c5338bdf231b2c0391d222cc920208d93e8ed72ff612b4a24db90e269b5"} Jul 29 08:22:47.277593 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:47.277318 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-khm2f" Jul 29 08:22:48.168874 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:48.168842 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hqzk8" Jul 29 08:22:48.187742 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:48.187688 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-khm2f" podStartSLOduration=66.279734565 podStartE2EDuration="1m8.187674123s" totalImagesPullingTime="1.907939558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:47.326394219 +0000 UTC m=+101.906075902" watchObservedRunningTime="2026-07-29 08:22:48.187674123 +0000 UTC m=+102.767355806" Jul 29 08:22:51.292414 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:51.292382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zbsm" event={"ID":"ff71c6ab-1aa9-4620-a002-c09e43bd308f","Type":"ContainerStarted","Data":"4f1975bb9bec5ac68419130e59237d38eb299f06ff611348563426365bcd7291"} Jul 29 08:22:54.302917 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:54.302885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zbsm" event={"ID":"ff71c6ab-1aa9-4620-a002-c09e43bd308f","Type":"ContainerStarted","Data":"c03cc17bda4ab5d505885960a6af558a81008b8eee58c8a3b3796f4d605beca9"} Jul 29 08:22:56.900366 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:56.900324 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c4f67556c-nt4x9"] Jul 29 08:22:57.284051 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:57.284015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-khm2f" Jul 29 08:22:58.318322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:58.318238 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zbsm" event={"ID":"ff71c6ab-1aa9-4620-a002-c09e43bd308f","Type":"ContainerStarted","Data":"4c6d1d3146141897b7f4828ec0cf1da73d5f9f31274b948a7c63e74e5fb408fe"} Jul 29 08:22:58.341376 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:22:58.341329 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5zbsm" podStartSLOduration=33.786807565 podStartE2EDuration="46.341315752s" totalImagesPullingTime="12.554508187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:22:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:22:58.33992751 +0000 UTC m=+112.919609194" watchObservedRunningTime="2026-07-29 08:22:58.341315752 +0000 UTC m=+112.920997434" Jul 29 08:23:06.906360 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:06.906324 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:23:09.435704 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.435672 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g25xf"] Jul 29 08:23:09.443819 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.443796 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.472341 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.472310 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 08:23:09.472470 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.472378 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 08:23:09.472548 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.472530 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 08:23:09.472607 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.472546 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 08:23:09.473163 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.473143 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mf9b9\"" Jul 29 08:23:09.473245 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.473205 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 08:23:09.484646 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.484629 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 08:23:09.499573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-wtmp\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499673 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499673 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-accelerators-collector-config\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499673 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-tls\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499800 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-root\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499800 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ph2h\" (UniqueName: \"kubernetes.io/projected/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-kube-api-access-9ph2h\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499800 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-sys\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499940 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-metrics-client-ca\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.499940 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.499850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-textfile\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.600884 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.600858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ph2h\" (UniqueName: \"kubernetes.io/projected/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-kube-api-access-9ph2h\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.600993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.600892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-sys\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.600993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.600921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-metrics-client-ca\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.600989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-sys\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-textfile\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-wtmp\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-accelerators-collector-config\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-tls\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-root\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-root\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-wtmp\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-textfile\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601352 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:23:09.601350 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 08:23:09.601758 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:23:09.601427 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-tls podName:a3e30b65-ffd5-47fd-bfcd-f33b6848a76b nodeName:}" failed. No retries permitted until 2026-07-29 08:23:10.101407155 +0000 UTC m=+124.681088821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-tls") pod "node-exporter-g25xf" (UID: "a3e30b65-ffd5-47fd-bfcd-f33b6848a76b") : secret "node-exporter-tls" not found Jul 29 08:23:09.601758 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-metrics-client-ca\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.601945 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.601894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-accelerators-collector-config\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.603719 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.603698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:09.657604 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:09.657573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ph2h\" (UniqueName: \"kubernetes.io/projected/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-kube-api-access-9ph2h\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:10.105195 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:10.105162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-tls\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:10.107518 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:10.107494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3e30b65-ffd5-47fd-bfcd-f33b6848a76b-node-exporter-tls\") pod \"node-exporter-g25xf\" (UID: \"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b\") " pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:10.354219 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:10.354196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g25xf" Jul 29 08:23:10.365638 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:23:10.365615 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e30b65_ffd5_47fd_bfcd_f33b6848a76b.slice/crio-5b151aef573a128683124ab99bdbabdbdcf1a7d131867821453850dd23858ae0 WatchSource:0}: Error finding container 5b151aef573a128683124ab99bdbabdbdcf1a7d131867821453850dd23858ae0: Status 404 returned error can't find the container with id 5b151aef573a128683124ab99bdbabdbdcf1a7d131867821453850dd23858ae0 Jul 29 08:23:11.365107 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:11.365077 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g25xf" event={"ID":"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b","Type":"ContainerStarted","Data":"5b151aef573a128683124ab99bdbabdbdcf1a7d131867821453850dd23858ae0"} Jul 29 08:23:12.371091 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:12.371060 2580 generic.go:356] "Generic (PLEG): container finished" podID="a3e30b65-ffd5-47fd-bfcd-f33b6848a76b" containerID="e3d61b2c4502afc307e842050416c5ebc5bd95c4a246bddb9f51a6e374073410" exitCode=0 Jul 29 08:23:12.371549 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:12.371109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g25xf" event={"ID":"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b","Type":"ContainerDied","Data":"e3d61b2c4502afc307e842050416c5ebc5bd95c4a246bddb9f51a6e374073410"} Jul 29 08:23:13.376526 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:13.376487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g25xf" event={"ID":"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b","Type":"ContainerStarted","Data":"400e72668eb2888a3e121cab2b3abab937e6b07c272f4cfb6f9c40c0f8177503"} Jul 29 08:23:13.376873 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:13.376542 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g25xf" event={"ID":"a3e30b65-ffd5-47fd-bfcd-f33b6848a76b","Type":"ContainerStarted","Data":"247e4270d8fb0af4fc576c3166fdb0a193c4ec6c73a791f01e85beef2e0a3e4d"} Jul 29 08:23:13.426499 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:13.426454 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g25xf" podStartSLOduration=3.3967969829999998 podStartE2EDuration="4.42641323s" totalImagesPullingTime="960.09766ms" totalInitContainerRuntime="69.518587ms" isStatefulPod=true podCreationTimestamp="2026-07-29 08:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:23:13.424690781 +0000 UTC m=+128.004372464" watchObservedRunningTime="2026-07-29 08:23:13.42641323 +0000 UTC m=+128.006094913" Jul 29 08:23:15.645498 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:15.645451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:23:15.648181 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:15.648157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf109907-5eb4-404b-8454-e87a1e2c5849-metrics-certs\") pod \"network-metrics-daemon-vmj9l\" (UID: \"bf109907-5eb4-404b-8454-e87a1e2c5849\") " pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:23:15.825029 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:15.825004 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7c8hq\"" Jul 29 08:23:15.833000 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:15.832982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vmj9l" Jul 29 08:23:15.995761 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:15.995732 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vmj9l"] Jul 29 08:23:15.999223 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:23:15.999182 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf109907_5eb4_404b_8454_e87a1e2c5849.slice/crio-96652625d5fff8b279fdcaebea0297d192bc2508d1fc1dfb991888545d8a243b WatchSource:0}: Error finding container 96652625d5fff8b279fdcaebea0297d192bc2508d1fc1dfb991888545d8a243b: Status 404 returned error can't find the container with id 96652625d5fff8b279fdcaebea0297d192bc2508d1fc1dfb991888545d8a243b Jul 29 08:23:16.387767 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:16.387702 2580 generic.go:356] "Generic (PLEG): container finished" podID="00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a" containerID="e7dfe04033cfc61cafa9b7341b3fbb691b6c770f3534714075f51edfe2758a79" exitCode=0 Jul 29 08:23:16.387905 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:16.387772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" event={"ID":"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a","Type":"ContainerDied","Data":"e7dfe04033cfc61cafa9b7341b3fbb691b6c770f3534714075f51edfe2758a79"} Jul 29 08:23:16.388088 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:16.388069 2580 scope.go:117] "RemoveContainer" containerID="e7dfe04033cfc61cafa9b7341b3fbb691b6c770f3534714075f51edfe2758a79" Jul 29 08:23:16.389009 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:16.388986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmj9l" event={"ID":"bf109907-5eb4-404b-8454-e87a1e2c5849","Type":"ContainerStarted","Data":"96652625d5fff8b279fdcaebea0297d192bc2508d1fc1dfb991888545d8a243b"} Jul 29 08:23:17.393845 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:17.393767 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-fcsxb" event={"ID":"00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a","Type":"ContainerStarted","Data":"e3173b3fb0f15f95fad5d62205c858276475dd03e40f8cd5ded00ce1337d60bf"} Jul 29 08:23:17.395463 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:17.395417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmj9l" event={"ID":"bf109907-5eb4-404b-8454-e87a1e2c5849","Type":"ContainerStarted","Data":"9b80b99cd6cefc7f6cb255e3383693fea9a20547b747d8fa33ef2f17aa6a559a"} Jul 29 08:23:17.395594 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:17.395479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vmj9l" event={"ID":"bf109907-5eb4-404b-8454-e87a1e2c5849","Type":"ContainerStarted","Data":"401580469f3466bd71d49fd356c8c55f5d7fba9b425d57e48031bf3cc6b3667c"} Jul 29 08:23:17.476852 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:17.476804 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vmj9l" podStartSLOduration=130.508556296 podStartE2EDuration="2m11.476790236s" totalImagesPullingTime="968.23394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:23:17.475952058 +0000 UTC m=+132.055633741" watchObservedRunningTime="2026-07-29 08:23:17.476790236 +0000 UTC m=+132.056471918" Jul 29 08:23:21.920125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:21.920087 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" podUID="b3ce430d-bc66-44d9-9331-6d4cb913a93f" containerName="registry" containerID="cri-o://c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47" gracePeriod=30 Jul 29 08:23:22.161999 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.161971 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:23:22.187320 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187267 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ce430d-bc66-44d9-9331-6d4cb913a93f-ca-trust-extracted\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.187320 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187298 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-trusted-ca\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.187517 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187335 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-bound-sa-token\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.187517 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187366 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqbnv\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-kube-api-access-vqbnv\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.187517 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187410 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-image-registry-private-configuration\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.187517 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187456 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-installation-pull-secrets\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.187517 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187485 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.187517 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187513 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-certificates\") pod \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\" (UID: \"b3ce430d-bc66-44d9-9331-6d4cb913a93f\") " Jul 29 08:23:22.188057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.187745 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:23:22.188279 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.188065 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:23:22.190048 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.190021 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:23:22.190344 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.190318 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-kube-api-access-vqbnv" (OuterVolumeSpecName: "kube-api-access-vqbnv") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "kube-api-access-vqbnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:23:22.190447 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.190338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:23:22.190447 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.190379 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:23:22.190684 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.190652 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:23:22.196513 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.196489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ce430d-bc66-44d9-9331-6d4cb913a93f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b3ce430d-bc66-44d9-9331-6d4cb913a93f" (UID: "b3ce430d-bc66-44d9-9331-6d4cb913a93f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:23:22.288474 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288451 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-bound-sa-token\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.288474 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288472 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqbnv\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-kube-api-access-vqbnv\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.288586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288483 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-image-registry-private-configuration\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.288586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288493 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ce430d-bc66-44d9-9331-6d4cb913a93f-installation-pull-secrets\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.288586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288501 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.288586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288509 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-registry-certificates\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.288586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288517 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ce430d-bc66-44d9-9331-6d4cb913a93f-ca-trust-extracted\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.288586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.288525 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ce430d-bc66-44d9-9331-6d4cb913a93f-trusted-ca\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:23:22.412936 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.412908 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3ce430d-bc66-44d9-9331-6d4cb913a93f" containerID="c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47" exitCode=0 Jul 29 08:23:22.413044 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.412966 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" Jul 29 08:23:22.413044 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.412972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" event={"ID":"b3ce430d-bc66-44d9-9331-6d4cb913a93f","Type":"ContainerDied","Data":"c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47"} Jul 29 08:23:22.413044 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.413013 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c4f67556c-nt4x9" event={"ID":"b3ce430d-bc66-44d9-9331-6d4cb913a93f","Type":"ContainerDied","Data":"a35f46acaa7ce4c0da36ff46193b9ea69f6299161c6090abe80f652fe49a69a5"} Jul 29 08:23:22.413044 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.413035 2580 scope.go:117] "RemoveContainer" containerID="c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47" Jul 29 08:23:22.425330 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.425312 2580 scope.go:117] "RemoveContainer" containerID="c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47" Jul 29 08:23:22.425664 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:23:22.425631 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47\": container with ID starting with c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47 not found: ID does not exist" containerID="c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47" Jul 29 08:23:22.425758 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.425665 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47"} err="failed to get container status \"c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47\": rpc error: code = NotFound desc = could not find container \"c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47\": container with ID starting with c0b03fff44ef90a643f4cb3cf64833a2bdb8484a9ae3a9cb08cb47ca386fce47 not found: ID does not exist" Jul 29 08:23:22.437626 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.437577 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c4f67556c-nt4x9"] Jul 29 08:23:22.442646 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:22.442622 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c4f67556c-nt4x9"] Jul 29 08:23:23.899743 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:23:23.899706 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ce430d-bc66-44d9-9331-6d4cb913a93f" path="/var/lib/kubelet/pods/b3ce430d-bc66-44d9-9331-6d4cb913a93f/volumes" Jul 29 08:24:33.983985 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.983943 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42"] Jul 29 08:24:33.984553 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.984198 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ce430d-bc66-44d9-9331-6d4cb913a93f" containerName="registry" Jul 29 08:24:33.984553 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.984213 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ce430d-bc66-44d9-9331-6d4cb913a93f" containerName="registry" Jul 29 08:24:33.984553 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.984263 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ce430d-bc66-44d9-9331-6d4cb913a93f" containerName="registry" Jul 29 08:24:33.986202 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.986171 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:33.989118 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.989091 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h4cbm\"" Jul 29 08:24:33.989118 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.989096 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 29 08:24:33.989317 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.989096 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 29 08:24:33.990128 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.990083 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 29 08:24:33.990278 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.990153 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 29 08:24:33.990278 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.990168 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 29 08:24:33.996342 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:33.996316 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 29 08:24:34.001974 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.001946 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42"] Jul 29 08:24:34.142337 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-metrics-client-ca\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.142337 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvksm\" (UniqueName: \"kubernetes.io/projected/6365423e-fc6c-48d9-b64f-68e18264b2c1-kube-api-access-tvksm\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.142533 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-serving-certs-ca-bundle\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.142533 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-secret-telemeter-client\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.142533 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-telemeter-client-tls\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.142533 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-federate-client-tls\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.142662 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.142662 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.142573 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.242840 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.242840 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-metrics-client-ca\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.242840 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvksm\" (UniqueName: \"kubernetes.io/projected/6365423e-fc6c-48d9-b64f-68e18264b2c1-kube-api-access-tvksm\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-serving-certs-ca-bundle\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-secret-telemeter-client\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-telemeter-client-tls\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-federate-client-tls\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.242974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243611 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.243588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-metrics-client-ca\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243725 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.243655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-serving-certs-ca-bundle\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.243991 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.243970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6365423e-fc6c-48d9-b64f-68e18264b2c1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.245647 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.245612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.245647 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.245618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-telemeter-client-tls\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.246033 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.246014 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-federate-client-tls\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.246033 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.246025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6365423e-fc6c-48d9-b64f-68e18264b2c1-secret-telemeter-client\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.252489 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.252471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvksm\" (UniqueName: \"kubernetes.io/projected/6365423e-fc6c-48d9-b64f-68e18264b2c1-kube-api-access-tvksm\") pod \"telemeter-client-5b6d9c9ddd-fdk42\" (UID: \"6365423e-fc6c-48d9-b64f-68e18264b2c1\") " pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.296404 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.296380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" Jul 29 08:24:34.427250 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.427222 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42"] Jul 29 08:24:34.430362 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:24:34.430340 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6365423e_fc6c_48d9_b64f_68e18264b2c1.slice/crio-d0981acb8f4bb62dfff9b4dac10742982936e7400683d2af2fb951e0cc0b1e2d WatchSource:0}: Error finding container d0981acb8f4bb62dfff9b4dac10742982936e7400683d2af2fb951e0cc0b1e2d: Status 404 returned error can't find the container with id d0981acb8f4bb62dfff9b4dac10742982936e7400683d2af2fb951e0cc0b1e2d Jul 29 08:24:34.668720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:34.668690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" event={"ID":"6365423e-fc6c-48d9-b64f-68e18264b2c1","Type":"ContainerStarted","Data":"d0981acb8f4bb62dfff9b4dac10742982936e7400683d2af2fb951e0cc0b1e2d"} Jul 29 08:24:36.676145 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:36.676068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" event={"ID":"6365423e-fc6c-48d9-b64f-68e18264b2c1","Type":"ContainerStarted","Data":"eefe848f854f52c6c9ee89c169f502f009148e4e40befa5fa98b040373be7a29"} Jul 29 08:24:37.681753 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:37.681723 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" event={"ID":"6365423e-fc6c-48d9-b64f-68e18264b2c1","Type":"ContainerStarted","Data":"65e7762448a82ee256c2913ec11587a9e32a7372fde92e486f8b25fd16e060e9"} Jul 29 08:24:37.682081 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:37.681779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" event={"ID":"6365423e-fc6c-48d9-b64f-68e18264b2c1","Type":"ContainerStarted","Data":"a5a2a86ada87439134b46082e98d6efae8d6ec176db139ced72c12542e3c0f45"} Jul 29 08:24:37.707185 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:24:37.707143 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b6d9c9ddd-fdk42" podStartSLOduration=1.712488344 podStartE2EDuration="4.707127585s" totalImagesPullingTime="2.994639241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:24:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:24:37.705521712 +0000 UTC m=+212.285203399" watchObservedRunningTime="2026-07-29 08:24:37.707127585 +0000 UTC m=+212.286809267" Jul 29 08:26:05.760943 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:26:05.760917 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 08:28:03.126800 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.126763 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl"] Jul 29 08:28:03.129684 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.129669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:28:03.132153 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.132133 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 29 08:28:03.133105 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.133090 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 29 08:28:03.133207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.133145 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-blczd\"" Jul 29 08:28:03.147333 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.147308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl"] Jul 29 08:28:03.265168 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.265139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlgrm\" (UniqueName: \"kubernetes.io/projected/fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8-kube-api-access-xlgrm\") pod \"feast-operator-controller-manager-64d67955f8-54hwl\" (UID: \"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8\") " pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:28:03.366327 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.366304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlgrm\" (UniqueName: \"kubernetes.io/projected/fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8-kube-api-access-xlgrm\") pod \"feast-operator-controller-manager-64d67955f8-54hwl\" (UID: \"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8\") " pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:28:03.375468 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.375421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlgrm\" (UniqueName: \"kubernetes.io/projected/fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8-kube-api-access-xlgrm\") pod \"feast-operator-controller-manager-64d67955f8-54hwl\" (UID: \"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8\") " pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:28:03.439006 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.438974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:28:03.562178 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.562154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl"] Jul 29 08:28:03.564558 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:28:03.564534 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd84a8bc_bdaf_4fb1_b6df_0139ccc801d8.slice/crio-11e84bf60e4f5742695ba9662e1c05264b606e6e759fd82944fd1956a32f3f70 WatchSource:0}: Error finding container 11e84bf60e4f5742695ba9662e1c05264b606e6e759fd82944fd1956a32f3f70: Status 404 returned error can't find the container with id 11e84bf60e4f5742695ba9662e1c05264b606e6e759fd82944fd1956a32f3f70 Jul 29 08:28:03.566402 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:03.566387 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:28:04.365751 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:04.365720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" event={"ID":"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8","Type":"ContainerStarted","Data":"11e84bf60e4f5742695ba9662e1c05264b606e6e759fd82944fd1956a32f3f70"} Jul 29 08:28:06.373873 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:06.373804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" event={"ID":"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8","Type":"ContainerStarted","Data":"b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5"} Jul 29 08:28:06.374219 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:06.373924 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:28:06.392339 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:06.392289 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" podStartSLOduration=0.853349498 podStartE2EDuration="3.392272905s" totalImagesPullingTime="2.538923407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:28:06.390990443 +0000 UTC m=+420.970672116" watchObservedRunningTime="2026-07-29 08:28:06.392272905 +0000 UTC m=+420.971954589" Jul 29 08:28:17.379854 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:28:17.379824 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:33:58.415849 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.415815 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-4qwfd"] Jul 29 08:33:58.418883 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.418867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4qwfd" Jul 29 08:33:58.421994 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.421966 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 29 08:33:58.422122 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.421995 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 29 08:33:58.422964 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.422947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-tl7r7\"" Jul 29 08:33:58.426531 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.426291 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4qwfd"] Jul 29 08:33:58.521085 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.521061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6m5h\" (UniqueName: \"kubernetes.io/projected/6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7-kube-api-access-p6m5h\") pod \"redis-69bbd798b6-4qwfd\" (UID: \"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7\") " pod="test-ns-feast/redis-69bbd798b6-4qwfd" Jul 29 08:33:58.621967 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.621939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6m5h\" (UniqueName: \"kubernetes.io/projected/6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7-kube-api-access-p6m5h\") pod \"redis-69bbd798b6-4qwfd\" (UID: \"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7\") " pod="test-ns-feast/redis-69bbd798b6-4qwfd" Jul 29 08:33:58.632652 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.632628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6m5h\" (UniqueName: \"kubernetes.io/projected/6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7-kube-api-access-p6m5h\") pod \"redis-69bbd798b6-4qwfd\" (UID: \"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7\") " pod="test-ns-feast/redis-69bbd798b6-4qwfd" Jul 29 08:33:58.729533 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.729476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4qwfd" Jul 29 08:33:58.871581 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.871552 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4qwfd"] Jul 29 08:33:58.874879 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:33:58.874846 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c49530b_28b2_4e53_ac3d_9c3aa1ebdfe7.slice/crio-613cc8a25749a4631ca49f7cdf68e88a4562e994e5a5c5c5e8eafc7ca1fac02e WatchSource:0}: Error finding container 613cc8a25749a4631ca49f7cdf68e88a4562e994e5a5c5c5e8eafc7ca1fac02e: Status 404 returned error can't find the container with id 613cc8a25749a4631ca49f7cdf68e88a4562e994e5a5c5c5e8eafc7ca1fac02e Jul 29 08:33:58.876730 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:58.876716 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:33:59.588230 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:33:59.588199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4qwfd" event={"ID":"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7","Type":"ContainerStarted","Data":"613cc8a25749a4631ca49f7cdf68e88a4562e994e5a5c5c5e8eafc7ca1fac02e"} Jul 29 08:34:00.593091 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:00.593011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4qwfd" event={"ID":"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7","Type":"ContainerStarted","Data":"bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376"} Jul 29 08:34:00.610827 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:00.610781 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-4qwfd" podStartSLOduration=1.185087509 podStartE2EDuration="2.61076634s" totalImagesPullingTime="1.425678831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:34:00.610069659 +0000 UTC m=+775.189751343" watchObservedRunningTime="2026-07-29 08:34:00.61076634 +0000 UTC m=+775.190448022" Jul 29 08:34:33.848529 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:33.848450 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-test-apply-t96b8"] Jul 29 08:34:33.851586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:33.851568 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-t96b8" Jul 29 08:34:33.853923 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:33.853900 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 29 08:34:33.854145 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:33.854132 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-c9j8h\"" Jul 29 08:34:33.860714 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:33.860689 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-t96b8"] Jul 29 08:34:33.952916 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:33.952896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8g2\" (UniqueName: \"kubernetes.io/projected/6353868b-3ca4-4989-b365-a8b777a81e5f-kube-api-access-4w8g2\") pod \"feast-test-apply-t96b8\" (UID: \"6353868b-3ca4-4989-b365-a8b777a81e5f\") " pod="test-ns-feast/feast-test-apply-t96b8" Jul 29 08:34:34.053447 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:34.053398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8g2\" (UniqueName: \"kubernetes.io/projected/6353868b-3ca4-4989-b365-a8b777a81e5f-kube-api-access-4w8g2\") pod \"feast-test-apply-t96b8\" (UID: \"6353868b-3ca4-4989-b365-a8b777a81e5f\") " pod="test-ns-feast/feast-test-apply-t96b8" Jul 29 08:34:34.062082 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:34.062058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8g2\" (UniqueName: \"kubernetes.io/projected/6353868b-3ca4-4989-b365-a8b777a81e5f-kube-api-access-4w8g2\") pod \"feast-test-apply-t96b8\" (UID: \"6353868b-3ca4-4989-b365-a8b777a81e5f\") " pod="test-ns-feast/feast-test-apply-t96b8" Jul 29 08:34:34.161067 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:34.161048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-t96b8" Jul 29 08:34:34.283942 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:34.283917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-t96b8"] Jul 29 08:34:34.288079 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:34:34.288045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6353868b_3ca4_4989_b365_a8b777a81e5f.slice/crio-f1d4016d9a5da73a82028522463e942fe7f15190c9a1abc7bde8ed9fa0ea6cbf WatchSource:0}: Error finding container f1d4016d9a5da73a82028522463e942fe7f15190c9a1abc7bde8ed9fa0ea6cbf: Status 404 returned error can't find the container with id f1d4016d9a5da73a82028522463e942fe7f15190c9a1abc7bde8ed9fa0ea6cbf Jul 29 08:34:34.727947 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:34.727919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-t96b8" event={"ID":"6353868b-3ca4-4989-b365-a8b777a81e5f","Type":"ContainerStarted","Data":"f1d4016d9a5da73a82028522463e942fe7f15190c9a1abc7bde8ed9fa0ea6cbf"} Jul 29 08:34:38.745974 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:38.745928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-t96b8" event={"ID":"6353868b-3ca4-4989-b365-a8b777a81e5f","Type":"ContainerStarted","Data":"e9f5e7b5c354c747e61cbd349da8d6448e407ea07178665cebc87f9d7e181cec"} Jul 29 08:34:45.774165 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:45.774130 2580 generic.go:356] "Generic (PLEG): container finished" podID="6353868b-3ca4-4989-b365-a8b777a81e5f" containerID="e9f5e7b5c354c747e61cbd349da8d6448e407ea07178665cebc87f9d7e181cec" exitCode=0 Jul 29 08:34:45.774606 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:45.774183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-t96b8" event={"ID":"6353868b-3ca4-4989-b365-a8b777a81e5f","Type":"ContainerDied","Data":"e9f5e7b5c354c747e61cbd349da8d6448e407ea07178665cebc87f9d7e181cec"} Jul 29 08:34:46.779352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:46.779318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-t96b8" event={"ID":"6353868b-3ca4-4989-b365-a8b777a81e5f","Type":"ContainerStarted","Data":"7d6c965cf85f86787556dfd1becefa378948e6dccf78de6f350c5ea82f541cab"} Jul 29 08:34:46.798023 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:46.797970 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-test-apply-t96b8" podStartSLOduration=2.918171957 podStartE2EDuration="13.797954465s" totalImagesPullingTime="4.298403288s" totalInitContainerRuntime="6.58137922s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:34:46.796408234 +0000 UTC m=+821.376089952" watchObservedRunningTime="2026-07-29 08:34:46.797954465 +0000 UTC m=+821.377636147" Jul 29 08:34:55.813704 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:55.813654 2580 generic.go:356] "Generic (PLEG): container finished" podID="6353868b-3ca4-4989-b365-a8b777a81e5f" containerID="7d6c965cf85f86787556dfd1becefa378948e6dccf78de6f350c5ea82f541cab" exitCode=0 Jul 29 08:34:55.814179 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:55.813735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-t96b8" event={"ID":"6353868b-3ca4-4989-b365-a8b777a81e5f","Type":"ContainerDied","Data":"7d6c965cf85f86787556dfd1becefa378948e6dccf78de6f350c5ea82f541cab"} Jul 29 08:34:56.955490 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:56.955464 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-t96b8" Jul 29 08:34:57.111580 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:57.111496 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w8g2\" (UniqueName: \"kubernetes.io/projected/6353868b-3ca4-4989-b365-a8b777a81e5f-kube-api-access-4w8g2\") pod \"6353868b-3ca4-4989-b365-a8b777a81e5f\" (UID: \"6353868b-3ca4-4989-b365-a8b777a81e5f\") " Jul 29 08:34:57.113700 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:57.113679 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6353868b-3ca4-4989-b365-a8b777a81e5f-kube-api-access-4w8g2" (OuterVolumeSpecName: "kube-api-access-4w8g2") pod "6353868b-3ca4-4989-b365-a8b777a81e5f" (UID: "6353868b-3ca4-4989-b365-a8b777a81e5f"). InnerVolumeSpecName "kube-api-access-4w8g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:34:57.212284 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:57.212258 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8g2\" (UniqueName: \"kubernetes.io/projected/6353868b-3ca4-4989-b365-a8b777a81e5f-kube-api-access-4w8g2\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:34:57.823470 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:57.823418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-t96b8" event={"ID":"6353868b-3ca4-4989-b365-a8b777a81e5f","Type":"ContainerDied","Data":"f1d4016d9a5da73a82028522463e942fe7f15190c9a1abc7bde8ed9fa0ea6cbf"} Jul 29 08:34:57.823470 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:57.823460 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-t96b8" Jul 29 08:34:57.823470 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:57.823476 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d4016d9a5da73a82028522463e942fe7f15190c9a1abc7bde8ed9fa0ea6cbf" Jul 29 08:34:58.001436 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:58.001417 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-t96b8_6353868b-3ca4-4989-b365-a8b777a81e5f/feast-1/0.log" Jul 29 08:34:58.007493 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:34:58.007475 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-t96b8_6353868b-3ca4-4989-b365-a8b777a81e5f/feast-0/0.log" Jul 29 08:35:34.988288 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:34.988252 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-test-apply-t96b8"] Jul 29 08:35:34.992514 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:34.992488 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-test-apply-t96b8"] Jul 29 08:35:35.006771 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.006749 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4qwfd"] Jul 29 08:35:35.006950 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.006906 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-4qwfd" podUID="6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7" containerName="redis" containerID="cri-o://bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376" gracePeriod=30 Jul 29 08:35:35.338996 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.338975 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4qwfd" Jul 29 08:35:35.357698 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.357671 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6m5h\" (UniqueName: \"kubernetes.io/projected/6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7-kube-api-access-p6m5h\") pod \"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7\" (UID: \"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7\") " Jul 29 08:35:35.359766 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.359737 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7-kube-api-access-p6m5h" (OuterVolumeSpecName: "kube-api-access-p6m5h") pod "6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7" (UID: "6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7"). InnerVolumeSpecName "kube-api-access-p6m5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:35:35.458134 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.458109 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6m5h\" (UniqueName: \"kubernetes.io/projected/6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7-kube-api-access-p6m5h\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:35:35.900518 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.900487 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6353868b-3ca4-4989-b365-a8b777a81e5f" path="/var/lib/kubelet/pods/6353868b-3ca4-4989-b365-a8b777a81e5f/volumes" Jul 29 08:35:35.953866 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.953842 2580 generic.go:356] "Generic (PLEG): container finished" podID="6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7" containerID="bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376" exitCode=0 Jul 29 08:35:35.953964 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.953886 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-4qwfd" Jul 29 08:35:35.953964 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.953908 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4qwfd" event={"ID":"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7","Type":"ContainerDied","Data":"bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376"} Jul 29 08:35:35.953964 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.953942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-4qwfd" event={"ID":"6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7","Type":"ContainerDied","Data":"613cc8a25749a4631ca49f7cdf68e88a4562e994e5a5c5c5e8eafc7ca1fac02e"} Jul 29 08:35:35.953964 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.953963 2580 scope.go:117] "RemoveContainer" containerID="bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376" Jul 29 08:35:35.961975 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.961960 2580 scope.go:117] "RemoveContainer" containerID="bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376" Jul 29 08:35:35.962235 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:35:35.962215 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376\": container with ID starting with bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376 not found: ID does not exist" containerID="bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376" Jul 29 08:35:35.962288 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.962242 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376"} err="failed to get container status \"bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376\": rpc error: code = NotFound desc = could not find container \"bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376\": container with ID starting with bfb493b80d3c49bd31f2caf9c0a856d17c5fbc9bee743b4fdf24766b48bd8376 not found: ID does not exist" Jul 29 08:35:35.970207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.970188 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4qwfd"] Jul 29 08:35:35.974283 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:35.974262 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-4qwfd"] Jul 29 08:35:37.899374 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:35:37.899339 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7" path="/var/lib/kubelet/pods/6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7/volumes" Jul 29 08:41:05.862770 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:05.862740 2580 scope.go:117] "RemoveContainer" containerID="7d6c965cf85f86787556dfd1becefa378948e6dccf78de6f350c5ea82f541cab" Jul 29 08:41:05.869803 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:05.869786 2580 scope.go:117] "RemoveContainer" containerID="e9f5e7b5c354c747e61cbd349da8d6448e407ea07178665cebc87f9d7e181cec" Jul 29 08:41:48.480913 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:48.480883 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl"] Jul 29 08:41:48.481345 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:48.481063 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" podUID="fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8" containerName="manager" containerID="cri-o://b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5" gracePeriod=10 Jul 29 08:41:48.743782 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:48.743758 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:41:48.907246 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:48.907215 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlgrm\" (UniqueName: \"kubernetes.io/projected/fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8-kube-api-access-xlgrm\") pod \"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8\" (UID: \"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8\") " Jul 29 08:41:48.909510 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:48.909489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8-kube-api-access-xlgrm" (OuterVolumeSpecName: "kube-api-access-xlgrm") pod "fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8" (UID: "fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8"). InnerVolumeSpecName "kube-api-access-xlgrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:41:49.008033 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.007951 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlgrm\" (UniqueName: \"kubernetes.io/projected/fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8-kube-api-access-xlgrm\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:41:49.160503 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.160474 2580 generic.go:356] "Generic (PLEG): container finished" podID="fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8" containerID="b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5" exitCode=0 Jul 29 08:41:49.160630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.160522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" event={"ID":"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8","Type":"ContainerDied","Data":"b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5"} Jul 29 08:41:49.160630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.160556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" event={"ID":"fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8","Type":"ContainerDied","Data":"11e84bf60e4f5742695ba9662e1c05264b606e6e759fd82944fd1956a32f3f70"} Jul 29 08:41:49.160630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.160577 2580 scope.go:117] "RemoveContainer" containerID="b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5" Jul 29 08:41:49.160630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.160522 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl" Jul 29 08:41:49.168882 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.168868 2580 scope.go:117] "RemoveContainer" containerID="b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5" Jul 29 08:41:49.169118 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:41:49.169102 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5\": container with ID starting with b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5 not found: ID does not exist" containerID="b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5" Jul 29 08:41:49.169161 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.169126 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5"} err="failed to get container status \"b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5\": rpc error: code = NotFound desc = could not find container \"b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5\": container with ID starting with b523d3d960c037ea22102156330cab78a132b8706cf32bb1e25d81b3e69fe7c5 not found: ID does not exist" Jul 29 08:41:49.182041 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.182018 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl"] Jul 29 08:41:49.186036 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.186019 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-64d67955f8-54hwl"] Jul 29 08:41:49.899248 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:41:49.899216 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8" path="/var/lib/kubelet/pods/fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8/volumes" Jul 29 08:42:00.485008 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.484933 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8"] Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485152 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6353868b-3ca4-4989-b365-a8b777a81e5f" containerName="feast-0" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485162 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6353868b-3ca4-4989-b365-a8b777a81e5f" containerName="feast-0" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485170 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6353868b-3ca4-4989-b365-a8b777a81e5f" containerName="feast-1" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485175 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6353868b-3ca4-4989-b365-a8b777a81e5f" containerName="feast-1" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485194 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7" containerName="redis" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485200 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7" containerName="redis" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485207 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8" containerName="manager" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485211 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8" containerName="manager" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485247 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd84a8bc-bdaf-4fb1-b6df-0139ccc801d8" containerName="manager" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485255 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6353868b-3ca4-4989-b365-a8b777a81e5f" containerName="feast-1" Jul 29 08:42:00.485335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.485262 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c49530b-28b2-4e53-ac3d-9c3aa1ebdfe7" containerName="redis" Jul 29 08:42:00.489114 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.489098 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:42:00.491690 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.491666 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 29 08:42:00.492637 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.492618 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-nh96n\"" Jul 29 08:42:00.492637 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.492632 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 29 08:42:00.500240 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.500221 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8"] Jul 29 08:42:00.580835 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.580811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57htk\" (UniqueName: \"kubernetes.io/projected/e28acc6c-78d9-4359-8ef2-bbf3b37647e8-kube-api-access-57htk\") pod \"feast-operator-controller-manager-7495bcd-jwdv8\" (UID: \"e28acc6c-78d9-4359-8ef2-bbf3b37647e8\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:42:00.681558 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.681524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57htk\" (UniqueName: \"kubernetes.io/projected/e28acc6c-78d9-4359-8ef2-bbf3b37647e8-kube-api-access-57htk\") pod \"feast-operator-controller-manager-7495bcd-jwdv8\" (UID: \"e28acc6c-78d9-4359-8ef2-bbf3b37647e8\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:42:00.692367 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.692343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57htk\" (UniqueName: \"kubernetes.io/projected/e28acc6c-78d9-4359-8ef2-bbf3b37647e8-kube-api-access-57htk\") pod \"feast-operator-controller-manager-7495bcd-jwdv8\" (UID: \"e28acc6c-78d9-4359-8ef2-bbf3b37647e8\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:42:00.798525 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.798473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:42:00.925271 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.925240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8"] Jul 29 08:42:00.928236 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:42:00.928208 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28acc6c_78d9_4359_8ef2_bbf3b37647e8.slice/crio-8c546a91a2b0932e49c9489a02176ce7838ec3788d749790e0016113d8a13c87 WatchSource:0}: Error finding container 8c546a91a2b0932e49c9489a02176ce7838ec3788d749790e0016113d8a13c87: Status 404 returned error can't find the container with id 8c546a91a2b0932e49c9489a02176ce7838ec3788d749790e0016113d8a13c87 Jul 29 08:42:00.930155 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:00.930139 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:42:01.208288 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:01.208256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" event={"ID":"e28acc6c-78d9-4359-8ef2-bbf3b37647e8","Type":"ContainerStarted","Data":"8c546a91a2b0932e49c9489a02176ce7838ec3788d749790e0016113d8a13c87"} Jul 29 08:42:04.219795 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:04.219756 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" event={"ID":"e28acc6c-78d9-4359-8ef2-bbf3b37647e8","Type":"ContainerStarted","Data":"2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9"} Jul 29 08:42:04.220168 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:04.219880 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:42:04.240212 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:04.240169 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" podStartSLOduration=1.862909417 podStartE2EDuration="4.240155671s" totalImagesPullingTime="2.377246254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:42:04.238940611 +0000 UTC m=+1258.818622292" watchObservedRunningTime="2026-07-29 08:42:04.240155671 +0000 UTC m=+1258.819837352" Jul 29 08:42:15.224475 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:15.224449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:42:17.167906 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.167872 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv"] Jul 29 08:42:17.176894 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.176866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.179705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.179554 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 29 08:42:17.179705 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.179632 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 29 08:42:17.180020 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.179998 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 29 08:42:17.180127 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.180068 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 29 08:42:17.180184 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.180005 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 29 08:42:17.180244 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.180230 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-fpkkg\"" Jul 29 08:42:17.180465 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.180449 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 29 08:42:17.190446 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.190412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv"] Jul 29 08:42:17.295354 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.295327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.295491 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.295365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzh2\" (UniqueName: \"kubernetes.io/projected/28da16c1-0198-4678-a705-9436fd927a16-kube-api-access-ptzh2\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.295491 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.295399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.295491 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.295457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28da16c1-0198-4678-a705-9436fd927a16-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.295610 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.295516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.295610 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.295566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.396096 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.396073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28da16c1-0198-4678-a705-9436fd927a16-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.396213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.396106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.396213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.396127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.396213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.396168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.396213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.396193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzh2\" (UniqueName: \"kubernetes.io/projected/28da16c1-0198-4678-a705-9436fd927a16-kube-api-access-ptzh2\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.396420 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.396222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.396420 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:42:17.396285 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 29 08:42:17.396420 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:42:17.396311 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 29 08:42:17.396420 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:42:17.396350 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls podName:28da16c1-0198-4678-a705-9436fd927a16 nodeName:}" failed. No retries permitted until 2026-07-29 08:42:17.896331151 +0000 UTC m=+1272.476012815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls") pod "feast-simple-feast-setup-777d6cc998-2k2gv" (UID: "28da16c1-0198-4678-a705-9436fd927a16") : secret "feast-simple-feast-setup-online-tls" not found Jul 29 08:42:17.396420 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:42:17.396367 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls podName:28da16c1-0198-4678-a705-9436fd927a16 nodeName:}" failed. No retries permitted until 2026-07-29 08:42:17.896358768 +0000 UTC m=+1272.476040434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls") pod "feast-simple-feast-setup-777d6cc998-2k2gv" (UID: "28da16c1-0198-4678-a705-9436fd927a16") : secret "feast-simple-feast-setup-ui-tls" not found Jul 29 08:42:17.396657 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.396522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28da16c1-0198-4678-a705-9436fd927a16-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.398972 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.398948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.399051 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.399010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.405201 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.405180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzh2\" (UniqueName: \"kubernetes.io/projected/28da16c1-0198-4678-a705-9436fd927a16-kube-api-access-ptzh2\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.900003 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.899976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.900171 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.900011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.902540 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.902513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:17.902654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:17.902585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-2k2gv\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:18.088322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:18.088294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:18.216852 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:18.216827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv"] Jul 29 08:42:18.218192 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:42:18.218167 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28da16c1_0198_4678_a705_9436fd927a16.slice/crio-0b17a68595b715226ff70f623732cb8f53043a1b134c80e50797e7e09c847247 WatchSource:0}: Error finding container 0b17a68595b715226ff70f623732cb8f53043a1b134c80e50797e7e09c847247: Status 404 returned error can't find the container with id 0b17a68595b715226ff70f623732cb8f53043a1b134c80e50797e7e09c847247 Jul 29 08:42:18.262767 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:18.262739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerStarted","Data":"0b17a68595b715226ff70f623732cb8f53043a1b134c80e50797e7e09c847247"} Jul 29 08:42:31.315657 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:31.315625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerStarted","Data":"da7fcd2750acc0dc9fa5d533158122e98349fa5621bfda20f0b043184ac7bf9c"} Jul 29 08:42:36.338149 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:36.338114 2580 generic.go:356] "Generic (PLEG): container finished" podID="28da16c1-0198-4678-a705-9436fd927a16" containerID="da7fcd2750acc0dc9fa5d533158122e98349fa5621bfda20f0b043184ac7bf9c" exitCode=0 Jul 29 08:42:36.338545 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:36.338154 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerDied","Data":"da7fcd2750acc0dc9fa5d533158122e98349fa5621bfda20f0b043184ac7bf9c"} Jul 29 08:42:37.343142 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:37.343103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerStarted","Data":"77ba985ac584563bc9f4ebe65d632b0f891d56657f94dab9e7ca49d53420bc06"} Jul 29 08:42:42.361845 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:42.361809 2580 generic.go:356] "Generic (PLEG): container finished" podID="28da16c1-0198-4678-a705-9436fd927a16" containerID="77ba985ac584563bc9f4ebe65d632b0f891d56657f94dab9e7ca49d53420bc06" exitCode=0 Jul 29 08:42:42.362303 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:42.361851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerDied","Data":"77ba985ac584563bc9f4ebe65d632b0f891d56657f94dab9e7ca49d53420bc06"} Jul 29 08:42:43.370720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:43.370681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerStarted","Data":"e6b571c8dd9e09656d8a664289498e100f9cd45fd4d173ff3332404b8f2e9edb"} Jul 29 08:42:43.371206 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:43.370741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerStarted","Data":"4698ff595767c1579d2bd8fa96dc8ce6112e835baba7bdcacfa0ce86777c3be5"} Jul 29 08:42:43.371206 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:43.370761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerStarted","Data":"792502f2a0960f558c8dc2a513be791981bcd70ce7884a97aacb94fb2baa9934"} Jul 29 08:42:43.371206 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:43.370778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerStarted","Data":"0633b9a88e9dfa36ff83b5c856bd1964667efbb72fc5f35b2c4bd6a8a3310ff0"} Jul 29 08:42:43.396855 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:43.396782 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podStartSLOduration=13.375354691 podStartE2EDuration="26.396760378s" totalImagesPullingTime="13.021405687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:42:43.395491614 +0000 UTC m=+1297.975173297" watchObservedRunningTime="2026-07-29 08:42:43.396760378 +0000 UTC m=+1297.976442060" Jul 29 08:42:45.088916 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.088875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:45.089473 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.089455 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:45.089607 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.089596 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:45.089923 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.089903 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:45.091260 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.091213 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.19:8816: connect: connection refused" Jul 29 08:42:45.091260 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.091231 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.19:6571: connect: connection refused" Jul 29 08:42:45.091260 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.091255 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" Jul 29 08:42:45.091510 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:45.091219 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" Jul 29 08:42:48.088593 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.088538 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.089084 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.088609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.089084 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.088630 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.089084 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.088648 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.089084 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.088753 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" Jul 29 08:42:48.089826 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.089806 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.089927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.089909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.094549 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.094528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.393663 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.393593 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.393887 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.393870 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:48.396385 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:48.396363 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:51.089667 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:51.089639 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:42:51.403796 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:42:51.403776 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:43:52.574756 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.574723 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv"] Jul 29 08:43:52.575510 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.575414 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="registry" containerID="cri-o://0633b9a88e9dfa36ff83b5c856bd1964667efbb72fc5f35b2c4bd6a8a3310ff0" gracePeriod=30 Jul 29 08:43:52.575678 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.575526 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" containerID="cri-o://792502f2a0960f558c8dc2a513be791981bcd70ce7884a97aacb94fb2baa9934" gracePeriod=30 Jul 29 08:43:52.575678 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.575535 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" containerID="cri-o://e6b571c8dd9e09656d8a664289498e100f9cd45fd4d173ff3332404b8f2e9edb" gracePeriod=30 Jul 29 08:43:52.575863 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.575824 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="offline" containerID="cri-o://4698ff595767c1579d2bd8fa96dc8ce6112e835baba7bdcacfa0ce86777c3be5" gracePeriod=30 Jul 29 08:43:52.874521 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.874311 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4"] Jul 29 08:43:52.878013 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.877986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:52.880716 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.880537 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-xwlpf\"" Jul 29 08:43:52.891101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.891079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4"] Jul 29 08:43:52.987542 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.987494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbhh\" (UniqueName: \"kubernetes.io/projected/986fe334-4457-4989-b434-5891c7f718bc-kube-api-access-ccbhh\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:52.987728 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.987581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/986fe334-4457-4989-b434-5891c7f718bc-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:52.987728 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.987660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:52.987728 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.987713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:52.987885 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.987747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:52.987885 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:52.987794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.088934 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.088825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.088934 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.088873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.089149 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.088981 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:43:53.089149 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.088982 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 29 08:43:53.089149 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.089041 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls podName:986fe334-4457-4989-b434-5891c7f718bc nodeName:}" failed. No retries permitted until 2026-07-29 08:43:53.589021073 +0000 UTC m=+1368.168702736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls") pod "feast-simple-feast-setup-777d6cc998-nbnx4" (UID: "986fe334-4457-4989-b434-5891c7f718bc") : secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:43:53.089149 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.089057 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls podName:986fe334-4457-4989-b434-5891c7f718bc nodeName:}" failed. No retries permitted until 2026-07-29 08:43:53.589049072 +0000 UTC m=+1368.168730733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls") pod "feast-simple-feast-setup-777d6cc998-nbnx4" (UID: "986fe334-4457-4989-b434-5891c7f718bc") : secret "feast-simple-feast-setup-registry-tls" not found Jul 29 08:43:53.089149 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.089074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.089149 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.089117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.089470 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.089168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbhh\" (UniqueName: \"kubernetes.io/projected/986fe334-4457-4989-b434-5891c7f718bc-kube-api-access-ccbhh\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.089470 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.089215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/986fe334-4457-4989-b434-5891c7f718bc-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.089470 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.089396 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 29 08:43:53.089635 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.089482 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls podName:986fe334-4457-4989-b434-5891c7f718bc nodeName:}" failed. No retries permitted until 2026-07-29 08:43:53.589467415 +0000 UTC m=+1368.169149093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls") pod "feast-simple-feast-setup-777d6cc998-nbnx4" (UID: "986fe334-4457-4989-b434-5891c7f718bc") : secret "feast-simple-feast-setup-ui-tls" not found Jul 29 08:43:53.089698 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.089637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/986fe334-4457-4989-b434-5891c7f718bc-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.092371 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.092345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.098037 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.098015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbhh\" (UniqueName: \"kubernetes.io/projected/986fe334-4457-4989-b434-5891c7f718bc-kube-api-access-ccbhh\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.594101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.594035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.594101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.594069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.594566 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.594112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.594566 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.594177 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:43:53.594566 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:43:53.594244 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls podName:986fe334-4457-4989-b434-5891c7f718bc nodeName:}" failed. No retries permitted until 2026-07-29 08:43:54.594227533 +0000 UTC m=+1369.173909196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls") pod "feast-simple-feast-setup-777d6cc998-nbnx4" (UID: "986fe334-4457-4989-b434-5891c7f718bc") : secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:43:53.596346 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.596318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.596471 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.596392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:53.606885 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.606861 2580 generic.go:356] "Generic (PLEG): container finished" podID="28da16c1-0198-4678-a705-9436fd927a16" containerID="e6b571c8dd9e09656d8a664289498e100f9cd45fd4d173ff3332404b8f2e9edb" exitCode=0 Jul 29 08:43:53.606987 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.606887 2580 generic.go:356] "Generic (PLEG): container finished" podID="28da16c1-0198-4678-a705-9436fd927a16" containerID="792502f2a0960f558c8dc2a513be791981bcd70ce7884a97aacb94fb2baa9934" exitCode=0 Jul 29 08:43:53.606987 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.606911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerDied","Data":"e6b571c8dd9e09656d8a664289498e100f9cd45fd4d173ff3332404b8f2e9edb"} Jul 29 08:43:53.606987 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:53.606975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerDied","Data":"792502f2a0960f558c8dc2a513be791981bcd70ce7884a97aacb94fb2baa9934"} Jul 29 08:43:54.600571 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:54.600534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:54.602849 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:54.602824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-nbnx4\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:54.691735 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:54.691708 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:43:54.819457 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:54.819411 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4"] Jul 29 08:43:54.820242 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:43:54.820218 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod986fe334_4457_4989_b434_5891c7f718bc.slice/crio-a483e0d8d6b4ccffcbaca49fd4ad0b754edc6cec4753c5c14591a3f13d533b2f WatchSource:0}: Error finding container a483e0d8d6b4ccffcbaca49fd4ad0b754edc6cec4753c5c14591a3f13d533b2f: Status 404 returned error can't find the container with id a483e0d8d6b4ccffcbaca49fd4ad0b754edc6cec4753c5c14591a3f13d533b2f Jul 29 08:43:55.615354 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:55.615316 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerStarted","Data":"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418"} Jul 29 08:43:55.615801 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:55.615372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerStarted","Data":"a483e0d8d6b4ccffcbaca49fd4ad0b754edc6cec4753c5c14591a3f13d533b2f"} Jul 29 08:43:58.393728 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:58.393686 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" Jul 29 08:43:59.631344 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:59.631265 2580 generic.go:356] "Generic (PLEG): container finished" podID="986fe334-4457-4989-b434-5891c7f718bc" containerID="29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418" exitCode=0 Jul 29 08:43:59.631692 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:43:59.631336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerDied","Data":"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418"} Jul 29 08:44:00.636714 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:00.636683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerStarted","Data":"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26"} Jul 29 08:44:01.404357 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:01.404318 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" Jul 29 08:44:05.656375 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:05.656331 2580 generic.go:356] "Generic (PLEG): container finished" podID="986fe334-4457-4989-b434-5891c7f718bc" containerID="12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26" exitCode=0 Jul 29 08:44:05.656750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:05.656404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerDied","Data":"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26"} Jul 29 08:44:06.666305 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.666254 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerStarted","Data":"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a"} Jul 29 08:44:06.666928 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.666318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerStarted","Data":"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797"} Jul 29 08:44:06.666928 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.666340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerStarted","Data":"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c"} Jul 29 08:44:06.666928 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.666361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerStarted","Data":"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e"} Jul 29 08:44:06.692163 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.692124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:06.692335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.692182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:06.692335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.692202 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:06.692335 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.692221 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:06.693917 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.693877 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.20:6567/health\": dial tcp 10.132.0.20:6567: connect: connection refused" Jul 29 08:44:06.693917 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.693892 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.20:8443: connect: connection refused" Jul 29 08:44:06.694091 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.693927 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.20:8816: connect: connection refused" Jul 29 08:44:06.694091 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.693887 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.20:6571: connect: connection refused" Jul 29 08:44:06.697878 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:06.697682 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podStartSLOduration=14.697665398 podStartE2EDuration="14.697665398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:44:06.696782454 +0000 UTC m=+1381.276464174" watchObservedRunningTime="2026-07-29 08:44:06.697665398 +0000 UTC m=+1381.277347080" Jul 29 08:44:08.394286 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:08.394243 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" Jul 29 08:44:09.693183 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:09.692927 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.20:8443: connect: connection refused" Jul 29 08:44:09.693183 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:09.692951 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.20:6571: connect: connection refused" Jul 29 08:44:09.693183 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:09.693072 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.20:6567/health\": dial tcp 10.132.0.20:6567: connect: connection refused" Jul 29 08:44:09.693183 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:09.693170 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.20:8816: connect: connection refused" Jul 29 08:44:11.404356 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:11.404304 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" Jul 29 08:44:12.692607 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.692577 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.693262 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.693244 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.693534 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.693515 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.693663 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.693546 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.693663 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.693572 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.693663 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.693588 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.694076 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.694052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.694193 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.694119 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.694193 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.694169 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:12.697826 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:12.697807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:13.700101 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:13.700068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:13.703329 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:13.703305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:44:18.393897 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:18.393849 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" Jul 29 08:44:18.394319 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:18.393995 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:44:21.404392 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:21.404345 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" Jul 29 08:44:21.404802 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:21.404507 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:44:22.735028 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:22.734992 2580 generic.go:356] "Generic (PLEG): container finished" podID="28da16c1-0198-4678-a705-9436fd927a16" containerID="4698ff595767c1579d2bd8fa96dc8ce6112e835baba7bdcacfa0ce86777c3be5" exitCode=137 Jul 29 08:44:22.735028 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:22.735018 2580 generic.go:356] "Generic (PLEG): container finished" podID="28da16c1-0198-4678-a705-9436fd927a16" containerID="0633b9a88e9dfa36ff83b5c856bd1964667efbb72fc5f35b2c4bd6a8a3310ff0" exitCode=137 Jul 29 08:44:22.735420 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:22.735060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerDied","Data":"4698ff595767c1579d2bd8fa96dc8ce6112e835baba7bdcacfa0ce86777c3be5"} Jul 29 08:44:22.735420 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:22.735105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerDied","Data":"0633b9a88e9dfa36ff83b5c856bd1964667efbb72fc5f35b2c4bd6a8a3310ff0"} Jul 29 08:44:23.213321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.213298 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:44:23.318940 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.318903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28da16c1-0198-4678-a705-9436fd927a16-feast-data\") pod \"28da16c1-0198-4678-a705-9436fd927a16\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " Jul 29 08:44:23.319125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.318960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptzh2\" (UniqueName: \"kubernetes.io/projected/28da16c1-0198-4678-a705-9436fd927a16-kube-api-access-ptzh2\") pod \"28da16c1-0198-4678-a705-9436fd927a16\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " Jul 29 08:44:23.319125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.318998 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls\") pod \"28da16c1-0198-4678-a705-9436fd927a16\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " Jul 29 08:44:23.319125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.319024 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls\") pod \"28da16c1-0198-4678-a705-9436fd927a16\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " Jul 29 08:44:23.319125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.319067 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-registry-tls\") pod \"28da16c1-0198-4678-a705-9436fd927a16\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " Jul 29 08:44:23.319125 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.319094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-offline-tls\") pod \"28da16c1-0198-4678-a705-9436fd927a16\" (UID: \"28da16c1-0198-4678-a705-9436fd927a16\") " Jul 29 08:44:23.319552 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.319523 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da16c1-0198-4678-a705-9436fd927a16-feast-data" (OuterVolumeSpecName: "feast-data") pod "28da16c1-0198-4678-a705-9436fd927a16" (UID: "28da16c1-0198-4678-a705-9436fd927a16"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:44:23.322153 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.322120 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "28da16c1-0198-4678-a705-9436fd927a16" (UID: "28da16c1-0198-4678-a705-9436fd927a16"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:23.322254 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.322160 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28da16c1-0198-4678-a705-9436fd927a16-kube-api-access-ptzh2" (OuterVolumeSpecName: "kube-api-access-ptzh2") pod "28da16c1-0198-4678-a705-9436fd927a16" (UID: "28da16c1-0198-4678-a705-9436fd927a16"). InnerVolumeSpecName "kube-api-access-ptzh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:44:23.322254 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.322167 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls" (OuterVolumeSpecName: "online-tls") pod "28da16c1-0198-4678-a705-9436fd927a16" (UID: "28da16c1-0198-4678-a705-9436fd927a16"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:23.322254 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.322180 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "28da16c1-0198-4678-a705-9436fd927a16" (UID: "28da16c1-0198-4678-a705-9436fd927a16"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:23.322254 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.322250 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "28da16c1-0198-4678-a705-9436fd927a16" (UID: "28da16c1-0198-4678-a705-9436fd927a16"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:44:23.420383 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.420354 2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/28da16c1-0198-4678-a705-9436fd927a16-feast-data\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:44:23.420383 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.420381 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptzh2\" (UniqueName: \"kubernetes.io/projected/28da16c1-0198-4678-a705-9436fd927a16-kube-api-access-ptzh2\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:44:23.420630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.420389 2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-online-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:44:23.420630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.420399 2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-ui-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:44:23.420630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.420406 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-registry-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:44:23.420630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.420413 2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/28da16c1-0198-4678-a705-9436fd927a16-offline-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:44:23.740866 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.740795 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" Jul 29 08:44:23.741255 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.740791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv" event={"ID":"28da16c1-0198-4678-a705-9436fd927a16","Type":"ContainerDied","Data":"0b17a68595b715226ff70f623732cb8f53043a1b134c80e50797e7e09c847247"} Jul 29 08:44:23.741255 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.740920 2580 scope.go:117] "RemoveContainer" containerID="e6b571c8dd9e09656d8a664289498e100f9cd45fd4d173ff3332404b8f2e9edb" Jul 29 08:44:23.754694 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.754675 2580 scope.go:117] "RemoveContainer" containerID="4698ff595767c1579d2bd8fa96dc8ce6112e835baba7bdcacfa0ce86777c3be5" Jul 29 08:44:23.761972 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.761950 2580 scope.go:117] "RemoveContainer" containerID="792502f2a0960f558c8dc2a513be791981bcd70ce7884a97aacb94fb2baa9934" Jul 29 08:44:23.768977 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.768961 2580 scope.go:117] "RemoveContainer" containerID="0633b9a88e9dfa36ff83b5c856bd1964667efbb72fc5f35b2c4bd6a8a3310ff0" Jul 29 08:44:23.775730 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.775711 2580 scope.go:117] "RemoveContainer" containerID="77ba985ac584563bc9f4ebe65d632b0f891d56657f94dab9e7ca49d53420bc06" Jul 29 08:44:23.779936 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.779916 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv"] Jul 29 08:44:23.782850 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.782825 2580 scope.go:117] "RemoveContainer" containerID="da7fcd2750acc0dc9fa5d533158122e98349fa5621bfda20f0b043184ac7bf9c" Jul 29 08:44:23.785860 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.785838 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-2k2gv"] Jul 29 08:44:23.900419 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:44:23.900381 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28da16c1-0198-4678-a705-9436fd927a16" path="/var/lib/kubelet/pods/28da16c1-0198-4678-a705-9436fd927a16/volumes" Jul 29 08:45:16.787481 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787452 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2"] Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787758 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787770 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787786 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="offline" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787794 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="offline" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787809 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="feast-init" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787815 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="feast-init" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787823 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787828 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787840 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="registry" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787848 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="registry" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787861 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="feast-apply" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787869 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="feast-apply" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787933 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="offline" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787941 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="ui" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787951 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="online" Jul 29 08:45:16.787956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.787960 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28da16c1-0198-4678-a705-9436fd927a16" containerName="registry" Jul 29 08:45:16.791981 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.791965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.794746 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.794724 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 29 08:45:16.794865 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.794765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-cs2fk\"" Jul 29 08:45:16.795012 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.794978 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 29 08:45:16.795068 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.795020 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 29 08:45:16.795068 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.795056 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:45:16.795161 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.794978 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 29 08:45:16.795254 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.795235 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 29 08:45:16.803876 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.803855 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2"] Jul 29 08:45:16.869074 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.869049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c380f573-573f-493d-9f55-e60b913f61ff-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.869207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.869095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/c380f573-573f-493d-9f55-e60b913f61ff-kube-api-access-mwhxk\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.869207 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.869171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c380f573-573f-493d-9f55-e60b913f61ff-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.869319 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.869225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.869319 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.869249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.869319 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.869264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.970257 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.970236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.970380 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.970270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.970380 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.970294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.970380 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.970323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c380f573-573f-493d-9f55-e60b913f61ff-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.970380 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.970355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/c380f573-573f-493d-9f55-e60b913f61ff-kube-api-access-mwhxk\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.970617 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:45:16.970396 2580 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 29 08:45:16.970617 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:45:16.970401 2580 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 29 08:45:16.970617 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:45:16.970484 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls podName:c380f573-573f-493d-9f55-e60b913f61ff nodeName:}" failed. No retries permitted until 2026-07-29 08:45:17.470463307 +0000 UTC m=+1452.050144975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls") pod "feast-simple-feast-remote-setup-db6955c6c-w5qm2" (UID: "c380f573-573f-493d-9f55-e60b913f61ff") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jul 29 08:45:16.970617 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.970405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c380f573-573f-493d-9f55-e60b913f61ff-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.970617 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:45:16.970504 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls podName:c380f573-573f-493d-9f55-e60b913f61ff nodeName:}" failed. No retries permitted until 2026-07-29 08:45:17.470494271 +0000 UTC m=+1452.050175933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls") pod "feast-simple-feast-remote-setup-db6955c6c-w5qm2" (UID: "c380f573-573f-493d-9f55-e60b913f61ff") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jul 29 08:45:16.970879 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.970858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c380f573-573f-493d-9f55-e60b913f61ff-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.971097 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.971078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c380f573-573f-493d-9f55-e60b913f61ff-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.973087 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.973070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:16.980242 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:16.980218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/c380f573-573f-493d-9f55-e60b913f61ff-kube-api-access-mwhxk\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:17.474759 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.474727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:17.474908 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.474776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:17.477075 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.477049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:17.477259 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.477240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-w5qm2\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:17.702170 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.702146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:17.830957 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.830876 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2"] Jul 29 08:45:17.833957 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:45:17.833921 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc380f573_573f_493d_9f55_e60b913f61ff.slice/crio-bd489204e8a1fa3e4858a0333ca26e99f7cf1e59ec4830f3b0db5847e5d98ec2 WatchSource:0}: Error finding container bd489204e8a1fa3e4858a0333ca26e99f7cf1e59ec4830f3b0db5847e5d98ec2: Status 404 returned error can't find the container with id bd489204e8a1fa3e4858a0333ca26e99f7cf1e59ec4830f3b0db5847e5d98ec2 Jul 29 08:45:17.923469 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.923421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerStarted","Data":"df7da24fa1be413ede150e81bb00e3db9b276de563db506fa6d775ae31e3654d"} Jul 29 08:45:17.923590 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:17.923487 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerStarted","Data":"bd489204e8a1fa3e4858a0333ca26e99f7cf1e59ec4830f3b0db5847e5d98ec2"} Jul 29 08:45:22.941797 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:22.941768 2580 generic.go:356] "Generic (PLEG): container finished" podID="c380f573-573f-493d-9f55-e60b913f61ff" containerID="df7da24fa1be413ede150e81bb00e3db9b276de563db506fa6d775ae31e3654d" exitCode=0 Jul 29 08:45:22.942138 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:22.941822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerDied","Data":"df7da24fa1be413ede150e81bb00e3db9b276de563db506fa6d775ae31e3654d"} Jul 29 08:45:23.947515 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:23.947471 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerStarted","Data":"8d2a97d290bac38d81e33c52aa08b81c0a001e0c0b63081cd103444b87b365db"} Jul 29 08:45:28.965557 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:28.965527 2580 generic.go:356] "Generic (PLEG): container finished" podID="c380f573-573f-493d-9f55-e60b913f61ff" containerID="8d2a97d290bac38d81e33c52aa08b81c0a001e0c0b63081cd103444b87b365db" exitCode=0 Jul 29 08:45:28.965895 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:28.965604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerDied","Data":"8d2a97d290bac38d81e33c52aa08b81c0a001e0c0b63081cd103444b87b365db"} Jul 29 08:45:29.973127 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:29.973096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerStarted","Data":"f74337ae66638e122f74af475f113c668539a3ab81c17a257e5afeabdb2a70f0"} Jul 29 08:45:29.973630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:29.973145 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerStarted","Data":"30f2c6f3ad4dcb88d5d240f1447dff7307479343ac9dee428ca7c77bc18e9780"} Jul 29 08:45:29.973630 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:29.973163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerStarted","Data":"5e1dd925e0b5ea61a9994de79ba10538bc606ef4f607b7fb8033c6fd0e9a3da8"} Jul 29 08:45:29.999742 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:29.999694 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podStartSLOduration=13.999674877 podStartE2EDuration="13.999674877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:45:29.998521955 +0000 UTC m=+1464.578203638" watchObservedRunningTime="2026-07-29 08:45:29.999674877 +0000 UTC m=+1464.579356560" Jul 29 08:45:32.703253 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:32.703215 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:32.703713 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:32.703280 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:32.703713 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:32.703299 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:32.705218 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:32.705185 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.21:8443: connect: connection refused" Jul 29 08:45:32.705352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:32.705214 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" probeResult="failure" output="Get \"https://10.132.0.21:6567/health\": dial tcp 10.132.0.21:6567: connect: connection refused" Jul 29 08:45:32.705352 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:32.705209 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.21:8816: connect: connection refused" Jul 29 08:45:35.704086 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.704051 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:35.704486 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.704224 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:35.704574 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.704558 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:35.704629 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.704595 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:35.704897 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.704877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:35.704995 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.704983 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:35.710365 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.710348 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:35.998150 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:35.998065 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:45:36.002488 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:45:36.002460 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:46:38.169683 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.169611 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2"] Jul 29 08:46:38.170189 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.169836 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" containerID="cri-o://5e1dd925e0b5ea61a9994de79ba10538bc606ef4f607b7fb8033c6fd0e9a3da8" gracePeriod=30 Jul 29 08:46:38.170189 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.169890 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" containerID="cri-o://f74337ae66638e122f74af475f113c668539a3ab81c17a257e5afeabdb2a70f0" gracePeriod=30 Jul 29 08:46:38.170189 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.169941 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="offline" containerID="cri-o://30f2c6f3ad4dcb88d5d240f1447dff7307479343ac9dee428ca7c77bc18e9780" gracePeriod=30 Jul 29 08:46:38.298413 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.298384 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4"] Jul 29 08:46:38.298695 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.298655 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="registry" containerID="cri-o://9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e" gracePeriod=30 Jul 29 08:46:38.298941 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.298738 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" containerID="cri-o://cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c" gracePeriod=30 Jul 29 08:46:38.299045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.298738 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="offline" containerID="cri-o://bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797" gracePeriod=30 Jul 29 08:46:38.299045 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:38.299010 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" containerID="cri-o://4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a" gracePeriod=30 Jul 29 08:46:39.210387 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.210360 2580 generic.go:356] "Generic (PLEG): container finished" podID="986fe334-4457-4989-b434-5891c7f718bc" containerID="4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a" exitCode=0 Jul 29 08:46:39.210387 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.210383 2580 generic.go:356] "Generic (PLEG): container finished" podID="986fe334-4457-4989-b434-5891c7f718bc" containerID="cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c" exitCode=0 Jul 29 08:46:39.210788 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.210446 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerDied","Data":"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a"} Jul 29 08:46:39.210788 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.210485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerDied","Data":"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c"} Jul 29 08:46:39.212868 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.212848 2580 generic.go:356] "Generic (PLEG): container finished" podID="c380f573-573f-493d-9f55-e60b913f61ff" containerID="f74337ae66638e122f74af475f113c668539a3ab81c17a257e5afeabdb2a70f0" exitCode=0 Jul 29 08:46:39.212868 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.212867 2580 generic.go:356] "Generic (PLEG): container finished" podID="c380f573-573f-493d-9f55-e60b913f61ff" containerID="5e1dd925e0b5ea61a9994de79ba10538bc606ef4f607b7fb8033c6fd0e9a3da8" exitCode=0 Jul 29 08:46:39.212983 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.212896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerDied","Data":"f74337ae66638e122f74af475f113c668539a3ab81c17a257e5afeabdb2a70f0"} Jul 29 08:46:39.212983 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:39.212924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerDied","Data":"5e1dd925e0b5ea61a9994de79ba10538bc606ef4f607b7fb8033c6fd0e9a3da8"} Jul 29 08:46:42.694120 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:42.694069 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.20:8443: connect: connection refused" Jul 29 08:46:43.700551 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:43.700512 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.20:6567/health\": dial tcp 10.132.0.20:6567: connect: connection refused" Jul 29 08:46:45.705354 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:45.705320 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.21:8443: connect: connection refused" Jul 29 08:46:45.998682 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:45.998605 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" probeResult="failure" output="Get \"https://10.132.0.21:6567/health\": dial tcp 10.132.0.21:6567: connect: connection refused" Jul 29 08:46:52.693863 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:52.693823 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.20:8443: connect: connection refused" Jul 29 08:46:53.700393 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:53.700354 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.20:6567/health\": dial tcp 10.132.0.20:6567: connect: connection refused" Jul 29 08:46:55.704875 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:55.704839 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.21:8443: connect: connection refused" Jul 29 08:46:55.999186 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:46:55.999108 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" probeResult="failure" output="Get \"https://10.132.0.21:6567/health\": dial tcp 10.132.0.21:6567: connect: connection refused" Jul 29 08:47:02.693822 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:02.693776 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.20:8443: connect: connection refused" Jul 29 08:47:02.694421 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:02.693929 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:47:03.700954 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:03.700920 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" probeResult="failure" output="Get \"https://10.132.0.20:6567/health\": dial tcp 10.132.0.20:6567: connect: connection refused" Jul 29 08:47:03.701289 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:03.701031 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:47:05.705566 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:05.705524 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.21:8443: connect: connection refused" Jul 29 08:47:05.705930 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:05.705645 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:47:05.998250 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:05.998176 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" probeResult="failure" output="Get \"https://10.132.0.21:6567/health\": dial tcp 10.132.0.21:6567: connect: connection refused" Jul 29 08:47:05.998381 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:05.998286 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:47:08.312663 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.312630 2580 generic.go:356] "Generic (PLEG): container finished" podID="c380f573-573f-493d-9f55-e60b913f61ff" containerID="30f2c6f3ad4dcb88d5d240f1447dff7307479343ac9dee428ca7c77bc18e9780" exitCode=137 Jul 29 08:47:08.313041 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.312697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerDied","Data":"30f2c6f3ad4dcb88d5d240f1447dff7307479343ac9dee428ca7c77bc18e9780"} Jul 29 08:47:08.813302 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.813273 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:47:08.909991 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.909962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/c380f573-573f-493d-9f55-e60b913f61ff-kube-api-access-mwhxk\") pod \"c380f573-573f-493d-9f55-e60b913f61ff\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " Jul 29 08:47:08.910158 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.910005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-online-tls\") pod \"c380f573-573f-493d-9f55-e60b913f61ff\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " Jul 29 08:47:08.910158 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.910037 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c380f573-573f-493d-9f55-e60b913f61ff-feast-data\") pod \"c380f573-573f-493d-9f55-e60b913f61ff\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " Jul 29 08:47:08.910158 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.910059 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls\") pod \"c380f573-573f-493d-9f55-e60b913f61ff\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " Jul 29 08:47:08.910158 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.910085 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls\") pod \"c380f573-573f-493d-9f55-e60b913f61ff\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " Jul 29 08:47:08.910158 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.910150 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c380f573-573f-493d-9f55-e60b913f61ff-registry-tls\") pod \"c380f573-573f-493d-9f55-e60b913f61ff\" (UID: \"c380f573-573f-493d-9f55-e60b913f61ff\") " Jul 29 08:47:08.910708 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.910680 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c380f573-573f-493d-9f55-e60b913f61ff-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c380f573-573f-493d-9f55-e60b913f61ff" (UID: "c380f573-573f-493d-9f55-e60b913f61ff"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:47:08.910962 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.910932 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c380f573-573f-493d-9f55-e60b913f61ff-feast-data" (OuterVolumeSpecName: "feast-data") pod "c380f573-573f-493d-9f55-e60b913f61ff" (UID: "c380f573-573f-493d-9f55-e60b913f61ff"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:47:08.912522 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.912496 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c380f573-573f-493d-9f55-e60b913f61ff" (UID: "c380f573-573f-493d-9f55-e60b913f61ff"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:08.912629 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.912606 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c380f573-573f-493d-9f55-e60b913f61ff" (UID: "c380f573-573f-493d-9f55-e60b913f61ff"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:08.912674 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.912657 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-online-tls" (OuterVolumeSpecName: "online-tls") pod "c380f573-573f-493d-9f55-e60b913f61ff" (UID: "c380f573-573f-493d-9f55-e60b913f61ff"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:08.912713 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.912670 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c380f573-573f-493d-9f55-e60b913f61ff-kube-api-access-mwhxk" (OuterVolumeSpecName: "kube-api-access-mwhxk") pod "c380f573-573f-493d-9f55-e60b913f61ff" (UID: "c380f573-573f-493d-9f55-e60b913f61ff"). InnerVolumeSpecName "kube-api-access-mwhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:47:08.948310 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:08.948289 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:47:09.010885 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.010857 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccbhh\" (UniqueName: \"kubernetes.io/projected/986fe334-4457-4989-b434-5891c7f718bc-kube-api-access-ccbhh\") pod \"986fe334-4457-4989-b434-5891c7f718bc\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " Jul 29 08:47:09.011052 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.010895 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls\") pod \"986fe334-4457-4989-b434-5891c7f718bc\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " Jul 29 08:47:09.011052 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.010940 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-online-tls\") pod \"986fe334-4457-4989-b434-5891c7f718bc\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " Jul 29 08:47:09.011052 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.010962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/986fe334-4457-4989-b434-5891c7f718bc-feast-data\") pod \"986fe334-4457-4989-b434-5891c7f718bc\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " Jul 29 08:47:09.011052 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.010981 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls\") pod \"986fe334-4457-4989-b434-5891c7f718bc\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " Jul 29 08:47:09.011052 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011004 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls\") pod \"986fe334-4457-4989-b434-5891c7f718bc\" (UID: \"986fe334-4457-4989-b434-5891c7f718bc\") " Jul 29 08:47:09.011303 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011157 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/c380f573-573f-493d-9f55-e60b913f61ff-kube-api-access-mwhxk\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.011303 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011174 2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-online-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.011303 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011187 2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c380f573-573f-493d-9f55-e60b913f61ff-feast-data\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.011303 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011198 2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-offline-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.011303 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011208 2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c380f573-573f-493d-9f55-e60b913f61ff-ui-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.011303 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011219 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c380f573-573f-493d-9f55-e60b913f61ff-registry-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.011717 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.011583 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986fe334-4457-4989-b434-5891c7f718bc-feast-data" (OuterVolumeSpecName: "feast-data") pod "986fe334-4457-4989-b434-5891c7f718bc" (UID: "986fe334-4457-4989-b434-5891c7f718bc"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:47:09.013343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.013323 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "986fe334-4457-4989-b434-5891c7f718bc" (UID: "986fe334-4457-4989-b434-5891c7f718bc"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:09.013646 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.013621 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "986fe334-4457-4989-b434-5891c7f718bc" (UID: "986fe334-4457-4989-b434-5891c7f718bc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:09.013748 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.013646 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "986fe334-4457-4989-b434-5891c7f718bc" (UID: "986fe334-4457-4989-b434-5891c7f718bc"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:09.013748 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.013713 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986fe334-4457-4989-b434-5891c7f718bc-kube-api-access-ccbhh" (OuterVolumeSpecName: "kube-api-access-ccbhh") pod "986fe334-4457-4989-b434-5891c7f718bc" (UID: "986fe334-4457-4989-b434-5891c7f718bc"). InnerVolumeSpecName "kube-api-access-ccbhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:47:09.013831 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.013748 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-online-tls" (OuterVolumeSpecName: "online-tls") pod "986fe334-4457-4989-b434-5891c7f718bc" (UID: "986fe334-4457-4989-b434-5891c7f718bc"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:47:09.111822 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.111787 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-registry-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.111822 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.111813 2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-online-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.111822 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.111821 2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/986fe334-4457-4989-b434-5891c7f718bc-feast-data\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.111822 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.111828 2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-offline-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.111822 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.111835 2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/986fe334-4457-4989-b434-5891c7f718bc-ui-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.112097 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.111842 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccbhh\" (UniqueName: \"kubernetes.io/projected/986fe334-4457-4989-b434-5891c7f718bc-kube-api-access-ccbhh\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:09.318635 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.318548 2580 generic.go:356] "Generic (PLEG): container finished" podID="986fe334-4457-4989-b434-5891c7f718bc" containerID="bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797" exitCode=137 Jul 29 08:47:09.318635 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.318578 2580 generic.go:356] "Generic (PLEG): container finished" podID="986fe334-4457-4989-b434-5891c7f718bc" containerID="9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e" exitCode=137 Jul 29 08:47:09.318635 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.318619 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" Jul 29 08:47:09.319216 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.318629 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerDied","Data":"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797"} Jul 29 08:47:09.319216 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.318675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerDied","Data":"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e"} Jul 29 08:47:09.319216 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.318694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4" event={"ID":"986fe334-4457-4989-b434-5891c7f718bc","Type":"ContainerDied","Data":"a483e0d8d6b4ccffcbaca49fd4ad0b754edc6cec4753c5c14591a3f13d533b2f"} Jul 29 08:47:09.319216 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.318713 2580 scope.go:117] "RemoveContainer" containerID="4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a" Jul 29 08:47:09.321291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.321274 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" Jul 29 08:47:09.321404 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.321272 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2" event={"ID":"c380f573-573f-493d-9f55-e60b913f61ff","Type":"ContainerDied","Data":"bd489204e8a1fa3e4858a0333ca26e99f7cf1e59ec4830f3b0db5847e5d98ec2"} Jul 29 08:47:09.327716 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.327360 2580 scope.go:117] "RemoveContainer" containerID="bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797" Jul 29 08:47:09.340057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.340040 2580 scope.go:117] "RemoveContainer" containerID="cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c" Jul 29 08:47:09.347396 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.347361 2580 scope.go:117] "RemoveContainer" containerID="9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e" Jul 29 08:47:09.348466 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.348445 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4"] Jul 29 08:47:09.353634 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.353616 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-nbnx4"] Jul 29 08:47:09.355176 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.355145 2580 scope.go:117] "RemoveContainer" containerID="12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26" Jul 29 08:47:09.362178 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.362160 2580 scope.go:117] "RemoveContainer" containerID="29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418" Jul 29 08:47:09.366540 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.366521 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2"] Jul 29 08:47:09.368917 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.368865 2580 scope.go:117] "RemoveContainer" containerID="4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a" Jul 29 08:47:09.369216 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:09.369183 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a\": container with ID starting with 4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a not found: ID does not exist" containerID="4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a" Jul 29 08:47:09.369321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.369271 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a"} err="failed to get container status \"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a\": rpc error: code = NotFound desc = could not find container \"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a\": container with ID starting with 4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a not found: ID does not exist" Jul 29 08:47:09.369392 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.369350 2580 scope.go:117] "RemoveContainer" containerID="bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797" Jul 29 08:47:09.369684 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:09.369664 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797\": container with ID starting with bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797 not found: ID does not exist" containerID="bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797" Jul 29 08:47:09.369783 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.369690 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797"} err="failed to get container status \"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797\": rpc error: code = NotFound desc = could not find container \"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797\": container with ID starting with bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797 not found: ID does not exist" Jul 29 08:47:09.369783 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.369710 2580 scope.go:117] "RemoveContainer" containerID="cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c" Jul 29 08:47:09.369995 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:09.369959 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c\": container with ID starting with cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c not found: ID does not exist" containerID="cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c" Jul 29 08:47:09.370082 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.369999 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c"} err="failed to get container status \"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c\": rpc error: code = NotFound desc = could not find container \"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c\": container with ID starting with cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c not found: ID does not exist" Jul 29 08:47:09.370082 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370020 2580 scope.go:117] "RemoveContainer" containerID="9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e" Jul 29 08:47:09.370278 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:09.370257 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e\": container with ID starting with 9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e not found: ID does not exist" containerID="9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e" Jul 29 08:47:09.370351 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370287 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e"} err="failed to get container status \"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e\": rpc error: code = NotFound desc = could not find container \"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e\": container with ID starting with 9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e not found: ID does not exist" Jul 29 08:47:09.370351 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370310 2580 scope.go:117] "RemoveContainer" containerID="12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26" Jul 29 08:47:09.370585 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:09.370563 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26\": container with ID starting with 12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26 not found: ID does not exist" containerID="12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26" Jul 29 08:47:09.370670 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370592 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26"} err="failed to get container status \"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26\": rpc error: code = NotFound desc = could not find container \"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26\": container with ID starting with 12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26 not found: ID does not exist" Jul 29 08:47:09.370670 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370611 2580 scope.go:117] "RemoveContainer" containerID="29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418" Jul 29 08:47:09.370861 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370844 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-w5qm2"] Jul 29 08:47:09.370897 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:09.370853 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418\": container with ID starting with 29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418 not found: ID does not exist" containerID="29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418" Jul 29 08:47:09.370897 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370873 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418"} err="failed to get container status \"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418\": rpc error: code = NotFound desc = could not find container \"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418\": container with ID starting with 29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418 not found: ID does not exist" Jul 29 08:47:09.370897 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.370889 2580 scope.go:117] "RemoveContainer" containerID="4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a" Jul 29 08:47:09.371124 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371106 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a"} err="failed to get container status \"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a\": rpc error: code = NotFound desc = could not find container \"4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a\": container with ID starting with 4074a33c0281516be6076c7f04d1d0832298c3c9f90a30455eb2a02d95954c2a not found: ID does not exist" Jul 29 08:47:09.371177 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371126 2580 scope.go:117] "RemoveContainer" containerID="bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797" Jul 29 08:47:09.371378 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371363 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797"} err="failed to get container status \"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797\": rpc error: code = NotFound desc = could not find container \"bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797\": container with ID starting with bc2cd3ddbc15ed3eacc10156637a94bd8b163769518c672402ebb1afbdfbf797 not found: ID does not exist" Jul 29 08:47:09.371426 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371380 2580 scope.go:117] "RemoveContainer" containerID="cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c" Jul 29 08:47:09.371619 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371600 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c"} err="failed to get container status \"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c\": rpc error: code = NotFound desc = could not find container \"cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c\": container with ID starting with cf57162e686e73252e42753ec55b785da07ceda30bb98c484161af100c051b6c not found: ID does not exist" Jul 29 08:47:09.371672 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371620 2580 scope.go:117] "RemoveContainer" containerID="9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e" Jul 29 08:47:09.371819 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371801 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e"} err="failed to get container status \"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e\": rpc error: code = NotFound desc = could not find container \"9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e\": container with ID starting with 9ca9080a0b239cea207659d518878c09cf0c839da5fb12e22a7c19c68a86df8e not found: ID does not exist" Jul 29 08:47:09.371887 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.371822 2580 scope.go:117] "RemoveContainer" containerID="12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26" Jul 29 08:47:09.372022 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.372004 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26"} err="failed to get container status \"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26\": rpc error: code = NotFound desc = could not find container \"12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26\": container with ID starting with 12756cf8dd05b02cebab6c6a2c99a109dfc5d1f69e651cfd74d8e4296f1efc26 not found: ID does not exist" Jul 29 08:47:09.372062 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.372023 2580 scope.go:117] "RemoveContainer" containerID="29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418" Jul 29 08:47:09.372217 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.372200 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418"} err="failed to get container status \"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418\": rpc error: code = NotFound desc = could not find container \"29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418\": container with ID starting with 29a3a48ddf37f6415ba4dd0692b5658776c639b3466755aad1129f87fbcde418 not found: ID does not exist" Jul 29 08:47:09.372263 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.372217 2580 scope.go:117] "RemoveContainer" containerID="f74337ae66638e122f74af475f113c668539a3ab81c17a257e5afeabdb2a70f0" Jul 29 08:47:09.379256 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.379238 2580 scope.go:117] "RemoveContainer" containerID="30f2c6f3ad4dcb88d5d240f1447dff7307479343ac9dee428ca7c77bc18e9780" Jul 29 08:47:09.386113 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.386103 2580 scope.go:117] "RemoveContainer" containerID="5e1dd925e0b5ea61a9994de79ba10538bc606ef4f607b7fb8033c6fd0e9a3da8" Jul 29 08:47:09.392719 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.392695 2580 scope.go:117] "RemoveContainer" containerID="8d2a97d290bac38d81e33c52aa08b81c0a001e0c0b63081cd103444b87b365db" Jul 29 08:47:09.399153 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.399140 2580 scope.go:117] "RemoveContainer" containerID="df7da24fa1be413ede150e81bb00e3db9b276de563db506fa6d775ae31e3654d" Jul 29 08:47:09.900534 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.900499 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="986fe334-4457-4989-b434-5891c7f718bc" path="/var/lib/kubelet/pods/986fe334-4457-4989-b434-5891c7f718bc/volumes" Jul 29 08:47:09.901069 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:09.901057 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c380f573-573f-493d-9f55-e60b913f61ff" path="/var/lib/kubelet/pods/c380f573-573f-493d-9f55-e60b913f61ff/volumes" Jul 29 08:47:27.379565 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:27.379526 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8"] Jul 29 08:47:27.379987 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:27.379752 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" podUID="e28acc6c-78d9-4359-8ef2-bbf3b37647e8" containerName="manager" containerID="cri-o://2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9" gracePeriod=10 Jul 29 08:47:27.621477 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:27.621454 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:47:27.749120 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:27.749089 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57htk\" (UniqueName: \"kubernetes.io/projected/e28acc6c-78d9-4359-8ef2-bbf3b37647e8-kube-api-access-57htk\") pod \"e28acc6c-78d9-4359-8ef2-bbf3b37647e8\" (UID: \"e28acc6c-78d9-4359-8ef2-bbf3b37647e8\") " Jul 29 08:47:27.751232 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:27.751195 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28acc6c-78d9-4359-8ef2-bbf3b37647e8-kube-api-access-57htk" (OuterVolumeSpecName: "kube-api-access-57htk") pod "e28acc6c-78d9-4359-8ef2-bbf3b37647e8" (UID: "e28acc6c-78d9-4359-8ef2-bbf3b37647e8"). InnerVolumeSpecName "kube-api-access-57htk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:47:27.850082 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:27.850054 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57htk\" (UniqueName: \"kubernetes.io/projected/e28acc6c-78d9-4359-8ef2-bbf3b37647e8-kube-api-access-57htk\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:47:28.387916 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.387886 2580 generic.go:356] "Generic (PLEG): container finished" podID="e28acc6c-78d9-4359-8ef2-bbf3b37647e8" containerID="2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9" exitCode=0 Jul 29 08:47:28.388300 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.387928 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" Jul 29 08:47:28.388300 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.387929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" event={"ID":"e28acc6c-78d9-4359-8ef2-bbf3b37647e8","Type":"ContainerDied","Data":"2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9"} Jul 29 08:47:28.388300 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.388008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8" event={"ID":"e28acc6c-78d9-4359-8ef2-bbf3b37647e8","Type":"ContainerDied","Data":"8c546a91a2b0932e49c9489a02176ce7838ec3788d749790e0016113d8a13c87"} Jul 29 08:47:28.388300 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.388038 2580 scope.go:117] "RemoveContainer" containerID="2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9" Jul 29 08:47:28.395424 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.395409 2580 scope.go:117] "RemoveContainer" containerID="2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9" Jul 29 08:47:28.395681 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:28.395659 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9\": container with ID starting with 2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9 not found: ID does not exist" containerID="2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9" Jul 29 08:47:28.395753 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.395689 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9"} err="failed to get container status \"2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9\": rpc error: code = NotFound desc = could not find container \"2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9\": container with ID starting with 2cbf4ca8f5981cc6a7d4a0cba6d0e080cfa12610b58520909c24325fc9b420d9 not found: ID does not exist" Jul 29 08:47:28.402975 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.402950 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8"] Jul 29 08:47:28.406942 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:28.406920 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-jwdv8"] Jul 29 08:47:29.899496 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:29.899464 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e28acc6c-78d9-4359-8ef2-bbf3b37647e8" path="/var/lib/kubelet/pods/e28acc6c-78d9-4359-8ef2-bbf3b37647e8/volumes" Jul 29 08:47:32.147088 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147049 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7"] Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147377 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="feast-init" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147392 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="feast-init" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147402 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147411 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147420 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="offline" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147427 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="offline" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147467 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="feast-apply" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147474 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="feast-apply" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147485 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="offline" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147492 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="offline" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147507 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147515 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147525 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="feast-apply" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147532 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="feast-apply" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147541 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147548 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147556 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="feast-init" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147563 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="feast-init" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147579 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e28acc6c-78d9-4359-8ef2-bbf3b37647e8" containerName="manager" Jul 29 08:47:32.147576 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147587 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e28acc6c-78d9-4359-8ef2-bbf3b37647e8" containerName="manager" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147595 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="registry" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147602 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="registry" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147610 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147618 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147686 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="online" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147698 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="offline" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147707 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e28acc6c-78d9-4359-8ef2-bbf3b37647e8" containerName="manager" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147716 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="ui" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147724 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="registry" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147732 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="offline" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147739 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="986fe334-4457-4989-b434-5891c7f718bc" containerName="ui" Jul 29 08:47:32.148452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.147747 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c380f573-573f-493d-9f55-e60b913f61ff" containerName="online" Jul 29 08:47:32.151946 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.151928 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" Jul 29 08:47:32.154667 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.154648 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 29 08:47:32.155614 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.155594 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 29 08:47:32.155614 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.155605 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-nh96n\"" Jul 29 08:47:32.158833 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.158710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7"] Jul 29 08:47:32.278713 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.278664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqs7r\" (UniqueName: \"kubernetes.io/projected/1d9f12d7-60fa-44e4-bbe4-95127ad32a3d-kube-api-access-wqs7r\") pod \"feast-operator-controller-manager-7495bcd-j84j7\" (UID: \"1d9f12d7-60fa-44e4-bbe4-95127ad32a3d\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" Jul 29 08:47:32.379573 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.379531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqs7r\" (UniqueName: \"kubernetes.io/projected/1d9f12d7-60fa-44e4-bbe4-95127ad32a3d-kube-api-access-wqs7r\") pod \"feast-operator-controller-manager-7495bcd-j84j7\" (UID: \"1d9f12d7-60fa-44e4-bbe4-95127ad32a3d\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" Jul 29 08:47:32.390291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.390260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqs7r\" (UniqueName: \"kubernetes.io/projected/1d9f12d7-60fa-44e4-bbe4-95127ad32a3d-kube-api-access-wqs7r\") pod \"feast-operator-controller-manager-7495bcd-j84j7\" (UID: \"1d9f12d7-60fa-44e4-bbe4-95127ad32a3d\") " pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" Jul 29 08:47:32.462798 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.462769 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" Jul 29 08:47:32.585113 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:32.585083 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7"] Jul 29 08:47:32.588742 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:47:32.588718 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9f12d7_60fa_44e4_bbe4_95127ad32a3d.slice/crio-70b634f732006abc1d3c12b0ee5e439758f94ba2ea5da7911c47b0b97cdc7216 WatchSource:0}: Error finding container 70b634f732006abc1d3c12b0ee5e439758f94ba2ea5da7911c47b0b97cdc7216: Status 404 returned error can't find the container with id 70b634f732006abc1d3c12b0ee5e439758f94ba2ea5da7911c47b0b97cdc7216 Jul 29 08:47:33.406953 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:33.406921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" event={"ID":"1d9f12d7-60fa-44e4-bbe4-95127ad32a3d","Type":"ContainerStarted","Data":"d04fb6f1d963cb859140cda874a76a8284eaac04317491d55cde0320a0d959e4"} Jul 29 08:47:33.407304 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:33.406968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" event={"ID":"1d9f12d7-60fa-44e4-bbe4-95127ad32a3d","Type":"ContainerStarted","Data":"70b634f732006abc1d3c12b0ee5e439758f94ba2ea5da7911c47b0b97cdc7216"} Jul 29 08:47:33.407304 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:33.407034 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" Jul 29 08:47:33.425155 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:33.425112 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" podStartSLOduration=1.42509994 podStartE2EDuration="1.42509994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:47:33.424572464 +0000 UTC m=+1588.004254158" watchObservedRunningTime="2026-07-29 08:47:33.42509994 +0000 UTC m=+1588.004781658" Jul 29 08:47:44.412815 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:44.412788 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7495bcd-j84j7" Jul 29 08:47:49.588362 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.588324 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt"] Jul 29 08:47:49.591755 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.591736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.594681 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.594654 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 29 08:47:49.594814 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.594732 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-t2gcb\"" Jul 29 08:47:49.594814 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.594775 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 29 08:47:49.594925 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.594854 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 29 08:47:49.595018 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.595003 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 29 08:47:49.595073 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.595007 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 29 08:47:49.595130 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.595033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 29 08:47:49.606648 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.606622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt"] Jul 29 08:47:49.688998 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.688973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1615e660-63a3-4e68-aceb-f4fd6cc7b835-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.688998 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.689000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.689171 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.689020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.689171 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.689039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4f45\" (UniqueName: \"kubernetes.io/projected/1615e660-63a3-4e68-aceb-f4fd6cc7b835-kube-api-access-n4f45\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.689171 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.689134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.689171 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.689168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.790113 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.790087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1615e660-63a3-4e68-aceb-f4fd6cc7b835-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.790329 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.790118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.790329 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.790139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.790329 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:49.790217 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 29 08:47:49.790329 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.790249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4f45\" (UniqueName: \"kubernetes.io/projected/1615e660-63a3-4e68-aceb-f4fd6cc7b835-kube-api-access-n4f45\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.790329 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:47:49.790266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls podName:1615e660-63a3-4e68-aceb-f4fd6cc7b835 nodeName:}" failed. No retries permitted until 2026-07-29 08:47:50.290246982 +0000 UTC m=+1604.869928649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls") pod "feast-simple-feast-setup-777d6cc998-wl2nt" (UID: "1615e660-63a3-4e68-aceb-f4fd6cc7b835") : secret "feast-simple-feast-setup-registry-tls" not found Jul 29 08:47:49.790329 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.790316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.790675 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.790357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.790675 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.790512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1615e660-63a3-4e68-aceb-f4fd6cc7b835-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.792777 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.792750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.792895 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.792778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.792954 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.792908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:49.801048 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:49.801028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4f45\" (UniqueName: \"kubernetes.io/projected/1615e660-63a3-4e68-aceb-f4fd6cc7b835-kube-api-access-n4f45\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:50.294231 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:50.294205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:50.296426 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:50.296407 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-wl2nt\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:50.502145 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:50.502125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:47:50.629613 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:50.629586 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt"] Jul 29 08:47:50.631268 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:47:50.631234 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1615e660_63a3_4e68_aceb_f4fd6cc7b835.slice/crio-0cd28dfb1e24b39acb159fd7837572ad0c1d2a497b2ee2168fc68ec5065b6260 WatchSource:0}: Error finding container 0cd28dfb1e24b39acb159fd7837572ad0c1d2a497b2ee2168fc68ec5065b6260: Status 404 returned error can't find the container with id 0cd28dfb1e24b39acb159fd7837572ad0c1d2a497b2ee2168fc68ec5065b6260 Jul 29 08:47:51.468524 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:51.468474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerStarted","Data":"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824"} Jul 29 08:47:51.468717 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:51.468540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerStarted","Data":"0cd28dfb1e24b39acb159fd7837572ad0c1d2a497b2ee2168fc68ec5065b6260"} Jul 29 08:47:55.483263 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:55.483233 2580 generic.go:356] "Generic (PLEG): container finished" podID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerID="0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824" exitCode=0 Jul 29 08:47:55.483678 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:55.483306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerDied","Data":"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824"} Jul 29 08:47:56.488167 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:47:56.488136 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerStarted","Data":"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960"} Jul 29 08:48:01.512392 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:01.512353 2580 generic.go:356] "Generic (PLEG): container finished" podID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerID="2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960" exitCode=0 Jul 29 08:48:01.512828 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:01.512427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerDied","Data":"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960"} Jul 29 08:48:02.521973 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:02.521934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerStarted","Data":"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6"} Jul 29 08:48:02.522555 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:02.522529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerStarted","Data":"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3"} Jul 29 08:48:02.522713 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:02.522694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerStarted","Data":"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38"} Jul 29 08:48:02.522825 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:02.522809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerStarted","Data":"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f"} Jul 29 08:48:02.549142 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:02.549089 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podStartSLOduration=13.549068847000001 podStartE2EDuration="13.549068847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:02.547003614 +0000 UTC m=+1617.126685297" watchObservedRunningTime="2026-07-29 08:48:02.549068847 +0000 UTC m=+1617.128750528" Jul 29 08:48:05.503051 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.502927 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:05.503051 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.502988 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:05.503051 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.503008 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:05.503051 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.503024 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:05.505106 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.504973 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.23:8816: connect: connection refused" Jul 29 08:48:05.505106 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.504993 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.23:6571: connect: connection refused" Jul 29 08:48:05.505106 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.505010 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Jul 29 08:48:05.505106 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:05.505088 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Jul 29 08:48:08.503167 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.503134 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.503767 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.503736 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.503932 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.503917 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.503979 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.503947 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.504014 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.504002 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.508358 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.508337 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.548410 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.548375 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.548410 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.548411 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.548589 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.548451 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.548839 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.548824 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.548896 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.548867 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:48:08.551170 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:48:08.551156 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:49:10.037625 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.037590 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt"] Jul 29 08:49:10.038182 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.037900 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="registry" containerID="cri-o://9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f" gracePeriod=30 Jul 29 08:49:10.038182 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.038075 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" containerID="cri-o://75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6" gracePeriod=30 Jul 29 08:49:10.038323 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.038204 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" containerID="cri-o://8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38" gracePeriod=30 Jul 29 08:49:10.038323 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.038254 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="offline" containerID="cri-o://acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3" gracePeriod=30 Jul 29 08:49:10.341703 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.341619 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q"] Jul 29 08:49:10.345322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.345302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.349292 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.349271 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-9kfnl\"" Jul 29 08:49:10.360455 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.358181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q"] Jul 29 08:49:10.422927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.422618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.422927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.422672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.422927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.422703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b59755b-1316-4779-be8a-98e93afe64ae-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.422927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.422762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.422927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.422787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.422927 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.422812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvbh\" (UniqueName: \"kubernetes.io/projected/1b59755b-1316-4779-be8a-98e93afe64ae-kube-api-access-8jvbh\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.523902 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.523863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.523902 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.523910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.524160 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.523936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvbh\" (UniqueName: \"kubernetes.io/projected/1b59755b-1316-4779-be8a-98e93afe64ae-kube-api-access-8jvbh\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.524160 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.523994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.524160 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.524018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.524160 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.524046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b59755b-1316-4779-be8a-98e93afe64ae-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.524467 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.524428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b59755b-1316-4779-be8a-98e93afe64ae-feast-data\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.539206 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:10.525595 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:49:10.539206 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:10.525683 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls podName:1b59755b-1316-4779-be8a-98e93afe64ae nodeName:}" failed. No retries permitted until 2026-07-29 08:49:11.025660616 +0000 UTC m=+1685.605342296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls") pod "feast-simple-feast-setup-777d6cc998-bsf5q" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae") : secret "feast-simple-feast-setup-offline-tls" not found Jul 29 08:49:10.539206 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:10.526529 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Jul 29 08:49:10.539206 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:10.526602 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls podName:1b59755b-1316-4779-be8a-98e93afe64ae nodeName:}" failed. No retries permitted until 2026-07-29 08:49:11.026582303 +0000 UTC m=+1685.606263970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls") pod "feast-simple-feast-setup-777d6cc998-bsf5q" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae") : secret "feast-simple-feast-setup-online-tls" not found Jul 29 08:49:10.539206 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:10.536684 2580 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 29 08:49:10.539206 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:10.536773 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls podName:1b59755b-1316-4779-be8a-98e93afe64ae nodeName:}" failed. No retries permitted until 2026-07-29 08:49:11.036743982 +0000 UTC m=+1685.616425658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls") pod "feast-simple-feast-setup-777d6cc998-bsf5q" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae") : secret "feast-simple-feast-setup-registry-tls" not found Jul 29 08:49:10.541819 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.541795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-ui-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.543941 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.543914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvbh\" (UniqueName: \"kubernetes.io/projected/1b59755b-1316-4779-be8a-98e93afe64ae-kube-api-access-8jvbh\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:10.759264 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.759236 2580 generic.go:356] "Generic (PLEG): container finished" podID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerID="75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6" exitCode=0 Jul 29 08:49:10.759264 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.759263 2580 generic.go:356] "Generic (PLEG): container finished" podID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerID="8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38" exitCode=0 Jul 29 08:49:10.759414 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.759304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerDied","Data":"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6"} Jul 29 08:49:10.759414 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:10.759348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerDied","Data":"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38"} Jul 29 08:49:11.028778 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.028746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:11.028778 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.028777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:11.031200 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.031176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:11.031353 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.031332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:11.129396 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.129370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:11.131368 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.131351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls\") pod \"feast-simple-feast-setup-777d6cc998-bsf5q\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:11.263036 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.263015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:11.393062 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.393039 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q"] Jul 29 08:49:11.394251 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:49:11.394232 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b59755b_1316_4779_be8a_98e93afe64ae.slice/crio-b0bcc4a5497d8143f35c463f2a987449a21b0bf327c06f2a2ffae07e54e5aa5a WatchSource:0}: Error finding container b0bcc4a5497d8143f35c463f2a987449a21b0bf327c06f2a2ffae07e54e5aa5a: Status 404 returned error can't find the container with id b0bcc4a5497d8143f35c463f2a987449a21b0bf327c06f2a2ffae07e54e5aa5a Jul 29 08:49:11.765277 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.765239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerStarted","Data":"6f01725ea38243baee81617557fb66f95d49497ec79d0bd083305863a0a44ab5"} Jul 29 08:49:11.765277 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:11.765284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerStarted","Data":"b0bcc4a5497d8143f35c463f2a987449a21b0bf327c06f2a2ffae07e54e5aa5a"} Jul 29 08:49:16.783204 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:16.783172 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b59755b-1316-4779-be8a-98e93afe64ae" containerID="6f01725ea38243baee81617557fb66f95d49497ec79d0bd083305863a0a44ab5" exitCode=0 Jul 29 08:49:16.783556 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:16.783228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerDied","Data":"6f01725ea38243baee81617557fb66f95d49497ec79d0bd083305863a0a44ab5"} Jul 29 08:49:17.788856 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:17.788826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerStarted","Data":"6792f6cdf3ae5c7d1f535bec1e7b3013a19dd7a03068a93ecf2488375a0d6c2a"} Jul 29 08:49:18.549241 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:18.549204 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Jul 29 08:49:18.549466 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:18.549294 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Jul 29 08:49:22.806979 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:22.806949 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b59755b-1316-4779-be8a-98e93afe64ae" containerID="6792f6cdf3ae5c7d1f535bec1e7b3013a19dd7a03068a93ecf2488375a0d6c2a" exitCode=0 Jul 29 08:49:22.807289 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:22.807026 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerDied","Data":"6792f6cdf3ae5c7d1f535bec1e7b3013a19dd7a03068a93ecf2488375a0d6c2a"} Jul 29 08:49:23.816896 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:23.816850 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerStarted","Data":"f64b53fb1310b69ea419e5a3eb719e1cb5655111d5e9e0e464188d4cb2941aae"} Jul 29 08:49:23.817388 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:23.816909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerStarted","Data":"a3d7ffacace613a21772e2fa4e24c89a98e0cc2a758d6384d8a68a7921befc39"} Jul 29 08:49:23.817388 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:23.816930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerStarted","Data":"edde2aa0a0b05b87a532600e058e92df2a4f63440efcb7a4e9c6380cc2c0c138"} Jul 29 08:49:23.817388 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:23.816948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerStarted","Data":"18139062fe99ed490704cd56336c84d6b4afc3ec0bf5c01a014fe68aa10efd9d"} Jul 29 08:49:23.841737 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:23.841669 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podStartSLOduration=13.841645793 podStartE2EDuration="13.841645793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:23.840347365 +0000 UTC m=+1698.420029087" watchObservedRunningTime="2026-07-29 08:49:23.841645793 +0000 UTC m=+1698.421327477" Jul 29 08:49:26.263464 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.263396 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:26.263933 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.263488 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:26.263933 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.263510 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:26.263933 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.263644 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:26.265384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.265330 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Jul 29 08:49:26.265384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.265348 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.24:8816: connect: connection refused" Jul 29 08:49:26.265636 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.265337 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Jul 29 08:49:26.265636 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:26.265340 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.24:6571: connect: connection refused" Jul 29 08:49:28.548891 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:28.548803 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Jul 29 08:49:28.548891 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:28.548821 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Jul 29 08:49:29.264694 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.264665 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.264862 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.264782 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.265016 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.264996 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.265330 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.265309 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.265411 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.265349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.265411 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.265367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.265589 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.265568 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.265654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.265640 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.265707 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.265696 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.269736 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.269717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.845569 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.845542 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:29.848636 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:29.848614 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:49:38.549501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:38.549456 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.23:8443: connect: connection refused" Jul 29 08:49:38.549880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:38.549543 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" probeResult="failure" output="Get \"https://10.132.0.23:6567/health\": dial tcp 10.132.0.23:6567: connect: connection refused" Jul 29 08:49:38.549880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:38.549577 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:49:38.549880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:38.549643 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:49:40.677813 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.677792 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:49:40.848975 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.848908 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4f45\" (UniqueName: \"kubernetes.io/projected/1615e660-63a3-4e68-aceb-f4fd6cc7b835-kube-api-access-n4f45\") pod \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " Jul 29 08:49:40.848975 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.848941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls\") pod \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " Jul 29 08:49:40.849151 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.848977 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-offline-tls\") pod \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " Jul 29 08:49:40.849151 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.849015 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-ui-tls\") pod \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " Jul 29 08:49:40.849151 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.849030 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-online-tls\") pod \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " Jul 29 08:49:40.849151 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.849055 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1615e660-63a3-4e68-aceb-f4fd6cc7b835-feast-data\") pod \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\" (UID: \"1615e660-63a3-4e68-aceb-f4fd6cc7b835\") " Jul 29 08:49:40.849693 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.849650 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1615e660-63a3-4e68-aceb-f4fd6cc7b835-feast-data" (OuterVolumeSpecName: "feast-data") pod "1615e660-63a3-4e68-aceb-f4fd6cc7b835" (UID: "1615e660-63a3-4e68-aceb-f4fd6cc7b835"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:49:40.851155 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.851134 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1615e660-63a3-4e68-aceb-f4fd6cc7b835-kube-api-access-n4f45" (OuterVolumeSpecName: "kube-api-access-n4f45") pod "1615e660-63a3-4e68-aceb-f4fd6cc7b835" (UID: "1615e660-63a3-4e68-aceb-f4fd6cc7b835"). InnerVolumeSpecName "kube-api-access-n4f45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:49:40.851590 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.851567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-online-tls" (OuterVolumeSpecName: "online-tls") pod "1615e660-63a3-4e68-aceb-f4fd6cc7b835" (UID: "1615e660-63a3-4e68-aceb-f4fd6cc7b835"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:49:40.851590 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.851579 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1615e660-63a3-4e68-aceb-f4fd6cc7b835" (UID: "1615e660-63a3-4e68-aceb-f4fd6cc7b835"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:49:40.851694 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.851618 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1615e660-63a3-4e68-aceb-f4fd6cc7b835" (UID: "1615e660-63a3-4e68-aceb-f4fd6cc7b835"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:49:40.851694 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.851634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1615e660-63a3-4e68-aceb-f4fd6cc7b835" (UID: "1615e660-63a3-4e68-aceb-f4fd6cc7b835"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:49:40.888466 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.888416 2580 generic.go:356] "Generic (PLEG): container finished" podID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerID="acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3" exitCode=137 Jul 29 08:49:40.888466 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.888456 2580 generic.go:356] "Generic (PLEG): container finished" podID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerID="9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f" exitCode=137 Jul 29 08:49:40.888583 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.888487 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" Jul 29 08:49:40.888583 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.888488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerDied","Data":"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3"} Jul 29 08:49:40.888657 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.888601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerDied","Data":"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f"} Jul 29 08:49:40.888657 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.888616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt" event={"ID":"1615e660-63a3-4e68-aceb-f4fd6cc7b835","Type":"ContainerDied","Data":"0cd28dfb1e24b39acb159fd7837572ad0c1d2a497b2ee2168fc68ec5065b6260"} Jul 29 08:49:40.888657 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.888634 2580 scope.go:117] "RemoveContainer" containerID="75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6" Jul 29 08:49:40.896426 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.896408 2580 scope.go:117] "RemoveContainer" containerID="acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3" Jul 29 08:49:40.903292 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.903276 2580 scope.go:117] "RemoveContainer" containerID="8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38" Jul 29 08:49:40.909900 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.909882 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt"] Jul 29 08:49:40.910833 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.910823 2580 scope.go:117] "RemoveContainer" containerID="9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f" Jul 29 08:49:40.915163 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.915143 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-wl2nt"] Jul 29 08:49:40.918175 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.918145 2580 scope.go:117] "RemoveContainer" containerID="2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960" Jul 29 08:49:40.924583 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.924572 2580 scope.go:117] "RemoveContainer" containerID="0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824" Jul 29 08:49:40.931579 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.931407 2580 scope.go:117] "RemoveContainer" containerID="75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6" Jul 29 08:49:40.931977 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:40.931871 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6\": container with ID starting with 75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6 not found: ID does not exist" containerID="75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6" Jul 29 08:49:40.931977 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.931900 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6"} err="failed to get container status \"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6\": rpc error: code = NotFound desc = could not find container \"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6\": container with ID starting with 75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6 not found: ID does not exist" Jul 29 08:49:40.931977 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.931921 2580 scope.go:117] "RemoveContainer" containerID="acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3" Jul 29 08:49:40.932365 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:40.932239 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3\": container with ID starting with acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3 not found: ID does not exist" containerID="acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3" Jul 29 08:49:40.932365 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.932264 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3"} err="failed to get container status \"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3\": rpc error: code = NotFound desc = could not find container \"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3\": container with ID starting with acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3 not found: ID does not exist" Jul 29 08:49:40.932365 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.932283 2580 scope.go:117] "RemoveContainer" containerID="8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38" Jul 29 08:49:40.932662 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:40.932641 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38\": container with ID starting with 8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38 not found: ID does not exist" containerID="8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38" Jul 29 08:49:40.932720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.932669 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38"} err="failed to get container status \"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38\": rpc error: code = NotFound desc = could not find container \"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38\": container with ID starting with 8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38 not found: ID does not exist" Jul 29 08:49:40.932720 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.932687 2580 scope.go:117] "RemoveContainer" containerID="9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f" Jul 29 08:49:40.932984 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:40.932965 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f\": container with ID starting with 9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f not found: ID does not exist" containerID="9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f" Jul 29 08:49:40.933057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.932992 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f"} err="failed to get container status \"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f\": rpc error: code = NotFound desc = could not find container \"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f\": container with ID starting with 9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f not found: ID does not exist" Jul 29 08:49:40.933057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.933012 2580 scope.go:117] "RemoveContainer" containerID="2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960" Jul 29 08:49:40.933576 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:40.933557 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960\": container with ID starting with 2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960 not found: ID does not exist" containerID="2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960" Jul 29 08:49:40.933650 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.933584 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960"} err="failed to get container status \"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960\": rpc error: code = NotFound desc = could not find container \"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960\": container with ID starting with 2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960 not found: ID does not exist" Jul 29 08:49:40.933650 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.933603 2580 scope.go:117] "RemoveContainer" containerID="0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824" Jul 29 08:49:40.933852 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:49:40.933828 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824\": container with ID starting with 0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824 not found: ID does not exist" containerID="0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824" Jul 29 08:49:40.933904 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.933859 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824"} err="failed to get container status \"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824\": rpc error: code = NotFound desc = could not find container \"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824\": container with ID starting with 0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824 not found: ID does not exist" Jul 29 08:49:40.933904 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.933876 2580 scope.go:117] "RemoveContainer" containerID="75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6" Jul 29 08:49:40.934142 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934124 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6"} err="failed to get container status \"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6\": rpc error: code = NotFound desc = could not find container \"75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6\": container with ID starting with 75cafb963849918d50539837813a07942d18bc61e8c0ef9cf36500f0a0c1ccc6 not found: ID does not exist" Jul 29 08:49:40.934193 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934144 2580 scope.go:117] "RemoveContainer" containerID="acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3" Jul 29 08:49:40.934343 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934324 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3"} err="failed to get container status \"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3\": rpc error: code = NotFound desc = could not find container \"acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3\": container with ID starting with acd11cbd5369cb761d5168a8459458adb70d5333d856ed356960712b654352d3 not found: ID does not exist" Jul 29 08:49:40.934409 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934344 2580 scope.go:117] "RemoveContainer" containerID="8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38" Jul 29 08:49:40.934580 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934566 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38"} err="failed to get container status \"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38\": rpc error: code = NotFound desc = could not find container \"8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38\": container with ID starting with 8969b5e410266c50b9e96eae1ac9d33ecc4fe27735a4feeb086cfea6d6cf7e38 not found: ID does not exist" Jul 29 08:49:40.934650 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934582 2580 scope.go:117] "RemoveContainer" containerID="9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f" Jul 29 08:49:40.934792 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934775 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f"} err="failed to get container status \"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f\": rpc error: code = NotFound desc = could not find container \"9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f\": container with ID starting with 9a1357fd7ee26bf914cd110e91d1619c187a7cd9478878fbd081c1bb37dd564f not found: ID does not exist" Jul 29 08:49:40.934849 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934794 2580 scope.go:117] "RemoveContainer" containerID="2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960" Jul 29 08:49:40.934974 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934959 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960"} err="failed to get container status \"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960\": rpc error: code = NotFound desc = could not find container \"2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960\": container with ID starting with 2d36e45f589a7121daa711db53d590f405a713c64ef01f814b9d070d9f915960 not found: ID does not exist" Jul 29 08:49:40.935035 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.934975 2580 scope.go:117] "RemoveContainer" containerID="0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824" Jul 29 08:49:40.935195 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.935177 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824"} err="failed to get container status \"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824\": rpc error: code = NotFound desc = could not find container \"0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824\": container with ID starting with 0c7a6fd7b59e6e8345060c7a36a507025877fb83d677fe85ede1e46713e08824 not found: ID does not exist" Jul 29 08:49:40.949955 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.949937 2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-ui-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:49:40.949955 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.949955 2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-online-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:49:40.950057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.949964 2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1615e660-63a3-4e68-aceb-f4fd6cc7b835-feast-data\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:49:40.950057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.949972 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4f45\" (UniqueName: \"kubernetes.io/projected/1615e660-63a3-4e68-aceb-f4fd6cc7b835-kube-api-access-n4f45\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:49:40.950057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.949979 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-registry-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:49:40.950057 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:40.949987 2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1615e660-63a3-4e68-aceb-f4fd6cc7b835-offline-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:49:41.902987 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:49:41.900155 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" path="/var/lib/kubelet/pods/1615e660-63a3-4e68-aceb-f4fd6cc7b835/volumes" Jul 29 08:50:31.792066 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792029 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt"] Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792358 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="feast-init" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792371 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="feast-init" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792385 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="offline" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792393 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="offline" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792402 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="feast-apply" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792410 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="feast-apply" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792445 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792453 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792469 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792477 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792488 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="registry" Jul 29 08:50:31.792497 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792495 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="registry" Jul 29 08:50:31.792835 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792550 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="registry" Jul 29 08:50:31.792835 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792562 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="ui" Jul 29 08:50:31.792835 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792570 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="online" Jul 29 08:50:31.792835 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.792578 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1615e660-63a3-4e68-aceb-f4fd6cc7b835" containerName="offline" Jul 29 08:50:31.796098 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.796072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:31.799627 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.799608 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 29 08:50:31.799784 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.799610 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:50:31.800012 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.799989 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 29 08:50:31.800126 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.800028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 29 08:50:31.800340 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.800323 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 29 08:50:31.800414 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.800373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-7cgtx\"" Jul 29 08:50:31.800612 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.800596 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 29 08:50:31.811753 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.811733 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt"] Jul 29 08:50:31.902733 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.902708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:31.902846 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.902742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/7aad2c76-7155-450e-b693-789734b64cdb-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:31.902846 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.902770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad2c76-7155-450e-b693-789734b64cdb-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:31.902952 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.902846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:31.902952 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.902879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dh8d\" (UniqueName: \"kubernetes.io/projected/7aad2c76-7155-450e-b693-789734b64cdb-kube-api-access-8dh8d\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:31.903043 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:31.902949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.003399 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.003375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.003519 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.003406 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dh8d\" (UniqueName: \"kubernetes.io/projected/7aad2c76-7155-450e-b693-789734b64cdb-kube-api-access-8dh8d\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.003519 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.003426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.003637 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.003511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.003637 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.003555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/7aad2c76-7155-450e-b693-789734b64cdb-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.003637 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.003592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad2c76-7155-450e-b693-789734b64cdb-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.003943 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.003918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad2c76-7155-450e-b693-789734b64cdb-feast-data\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.004218 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.004202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/7aad2c76-7155-450e-b693-789734b64cdb-registry-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.005868 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.005850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-ui-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.005954 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.005904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-online-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.006123 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.006107 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-offline-tls\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.013594 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.013568 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dh8d\" (UniqueName: \"kubernetes.io/projected/7aad2c76-7155-450e-b693-789734b64cdb-kube-api-access-8dh8d\") pod \"feast-simple-feast-remote-setup-db6955c6c-rg5nt\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.105463 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.105393 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:32.257461 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:32.257424 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt"] Jul 29 08:50:32.259327 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:50:32.259306 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aad2c76_7155_450e_b693_789734b64cdb.slice/crio-077499a94d3f440b614cdd3d22ddcb027a99b2ef731813516342c346609c5dfb WatchSource:0}: Error finding container 077499a94d3f440b614cdd3d22ddcb027a99b2ef731813516342c346609c5dfb: Status 404 returned error can't find the container with id 077499a94d3f440b614cdd3d22ddcb027a99b2ef731813516342c346609c5dfb Jul 29 08:50:33.060874 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:33.060843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerStarted","Data":"196c57b658ba265583beab0664da79a66c66341234870e9e6e683ec9c29dce52"} Jul 29 08:50:33.061205 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:33.060887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerStarted","Data":"077499a94d3f440b614cdd3d22ddcb027a99b2ef731813516342c346609c5dfb"} Jul 29 08:50:37.075723 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:37.075652 2580 generic.go:356] "Generic (PLEG): container finished" podID="7aad2c76-7155-450e-b693-789734b64cdb" containerID="196c57b658ba265583beab0664da79a66c66341234870e9e6e683ec9c29dce52" exitCode=0 Jul 29 08:50:37.075723 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:37.075690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerDied","Data":"196c57b658ba265583beab0664da79a66c66341234870e9e6e683ec9c29dce52"} Jul 29 08:50:38.080653 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:38.080621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerStarted","Data":"de25f495194d90a6cde11475b1b8bf7890915a891dd093b68d0fdbc6b0f01b08"} Jul 29 08:50:43.099747 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:43.099713 2580 generic.go:356] "Generic (PLEG): container finished" podID="7aad2c76-7155-450e-b693-789734b64cdb" containerID="de25f495194d90a6cde11475b1b8bf7890915a891dd093b68d0fdbc6b0f01b08" exitCode=0 Jul 29 08:50:43.100104 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:43.099754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerDied","Data":"de25f495194d90a6cde11475b1b8bf7890915a891dd093b68d0fdbc6b0f01b08"} Jul 29 08:50:44.108484 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:44.108420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerStarted","Data":"4f7d7a1996a5f8983273a45706d77bd34b0396d5197aba844428119ba2bbd03d"} Jul 29 08:50:44.108854 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:44.108507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerStarted","Data":"1a067435baf8eaf62988d190785b70c97df3a34060a868286ec76e720adc8047"} Jul 29 08:50:44.108854 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:44.108529 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerStarted","Data":"a70dd1d1b3f879764ab5b0590122ca9dff523193e6c9997748c8cdc256a978ae"} Jul 29 08:50:44.139265 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:44.139208 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podStartSLOduration=13.139188621 podStartE2EDuration="13.139188621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:50:44.138120941 +0000 UTC m=+1778.717802623" watchObservedRunningTime="2026-07-29 08:50:44.139188621 +0000 UTC m=+1778.718870303" Jul 29 08:50:47.105838 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:47.105800 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:47.106384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:47.106348 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:47.106384 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:47.106379 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:47.107626 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:47.107594 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.25:8816: connect: connection refused" Jul 29 08:50:47.107755 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:47.107630 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Jul 29 08:50:47.107755 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:47.107631 2580 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Jul 29 08:50:50.107074 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.107041 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.107536 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.107478 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.107536 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.107514 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.107817 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.107799 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.113174 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.113156 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.132185 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.132165 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.132291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.132194 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.132597 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.132580 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:50:50.135945 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:50:50.135925 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:51:52.865254 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:52.865222 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt"] Jul 29 08:51:52.865787 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:52.865496 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" containerID="cri-o://a70dd1d1b3f879764ab5b0590122ca9dff523193e6c9997748c8cdc256a978ae" gracePeriod=30 Jul 29 08:51:52.865787 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:52.865594 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" containerID="cri-o://4f7d7a1996a5f8983273a45706d77bd34b0396d5197aba844428119ba2bbd03d" gracePeriod=30 Jul 29 08:51:52.865787 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:52.865601 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="offline" containerID="cri-o://1a067435baf8eaf62988d190785b70c97df3a34060a868286ec76e720adc8047" gracePeriod=30 Jul 29 08:51:53.001506 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:53.001467 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q"] Jul 29 08:51:53.001813 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:53.001779 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="registry" containerID="cri-o://18139062fe99ed490704cd56336c84d6b4afc3ec0bf5c01a014fe68aa10efd9d" gracePeriod=30 Jul 29 08:51:53.002056 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:53.001844 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" containerID="cri-o://edde2aa0a0b05b87a532600e058e92df2a4f63440efcb7a4e9c6380cc2c0c138" gracePeriod=30 Jul 29 08:51:53.002171 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:53.001819 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="offline" containerID="cri-o://a3d7ffacace613a21772e2fa4e24c89a98e0cc2a758d6384d8a68a7921befc39" gracePeriod=30 Jul 29 08:51:53.002234 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:53.001859 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" containerID="cri-o://f64b53fb1310b69ea419e5a3eb719e1cb5655111d5e9e0e464188d4cb2941aae" gracePeriod=30 Jul 29 08:51:54.346786 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.346759 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b59755b-1316-4779-be8a-98e93afe64ae" containerID="f64b53fb1310b69ea419e5a3eb719e1cb5655111d5e9e0e464188d4cb2941aae" exitCode=0 Jul 29 08:51:54.346786 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.346783 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b59755b-1316-4779-be8a-98e93afe64ae" containerID="edde2aa0a0b05b87a532600e058e92df2a4f63440efcb7a4e9c6380cc2c0c138" exitCode=0 Jul 29 08:51:54.347282 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.346830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerDied","Data":"f64b53fb1310b69ea419e5a3eb719e1cb5655111d5e9e0e464188d4cb2941aae"} Jul 29 08:51:54.347282 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.346872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerDied","Data":"edde2aa0a0b05b87a532600e058e92df2a4f63440efcb7a4e9c6380cc2c0c138"} Jul 29 08:51:54.349248 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.349227 2580 generic.go:356] "Generic (PLEG): container finished" podID="7aad2c76-7155-450e-b693-789734b64cdb" containerID="4f7d7a1996a5f8983273a45706d77bd34b0396d5197aba844428119ba2bbd03d" exitCode=0 Jul 29 08:51:54.349248 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.349247 2580 generic.go:356] "Generic (PLEG): container finished" podID="7aad2c76-7155-450e-b693-789734b64cdb" containerID="a70dd1d1b3f879764ab5b0590122ca9dff523193e6c9997748c8cdc256a978ae" exitCode=0 Jul 29 08:51:54.349376 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.349268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerDied","Data":"4f7d7a1996a5f8983273a45706d77bd34b0396d5197aba844428119ba2bbd03d"} Jul 29 08:51:54.349376 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:54.349292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerDied","Data":"a70dd1d1b3f879764ab5b0590122ca9dff523193e6c9997748c8cdc256a978ae"} Jul 29 08:51:59.265664 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:59.265624 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Jul 29 08:51:59.846546 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:51:59.846509 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Jul 29 08:52:00.108135 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:00.108051 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Jul 29 08:52:00.132357 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:00.132325 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Jul 29 08:52:09.266115 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:09.266072 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Jul 29 08:52:09.846674 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:09.846639 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Jul 29 08:52:10.108103 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:10.108031 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Jul 29 08:52:10.132488 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:10.132403 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Jul 29 08:52:19.266342 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:19.266285 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.24:8443: connect: connection refused" Jul 29 08:52:19.266933 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:19.266474 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:52:19.846452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:19.846408 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" probeResult="failure" output="Get \"https://10.132.0.24:6567/health\": dial tcp 10.132.0.24:6567: connect: connection refused" Jul 29 08:52:19.846620 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:19.846527 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:52:20.108111 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:20.108028 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" Jul 29 08:52:20.108253 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:20.108156 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:52:20.133191 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:20.133152 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" Jul 29 08:52:20.133333 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:20.133258 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:52:23.455396 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.455369 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b59755b-1316-4779-be8a-98e93afe64ae" containerID="a3d7ffacace613a21772e2fa4e24c89a98e0cc2a758d6384d8a68a7921befc39" exitCode=137 Jul 29 08:52:23.455396 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.455394 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b59755b-1316-4779-be8a-98e93afe64ae" containerID="18139062fe99ed490704cd56336c84d6b4afc3ec0bf5c01a014fe68aa10efd9d" exitCode=137 Jul 29 08:52:23.455840 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.455459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerDied","Data":"a3d7ffacace613a21772e2fa4e24c89a98e0cc2a758d6384d8a68a7921befc39"} Jul 29 08:52:23.455840 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.455502 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerDied","Data":"18139062fe99ed490704cd56336c84d6b4afc3ec0bf5c01a014fe68aa10efd9d"} Jul 29 08:52:23.458398 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.458373 2580 generic.go:356] "Generic (PLEG): container finished" podID="7aad2c76-7155-450e-b693-789734b64cdb" containerID="1a067435baf8eaf62988d190785b70c97df3a34060a868286ec76e720adc8047" exitCode=137 Jul 29 08:52:23.458535 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.458460 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerDied","Data":"1a067435baf8eaf62988d190785b70c97df3a34060a868286ec76e720adc8047"} Jul 29 08:52:23.504946 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.504930 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:52:23.555566 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.555541 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad2c76-7155-450e-b693-789734b64cdb-feast-data\") pod \"7aad2c76-7155-450e-b693-789734b64cdb\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " Jul 29 08:52:23.555717 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.555589 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/7aad2c76-7155-450e-b693-789734b64cdb-registry-tls\") pod \"7aad2c76-7155-450e-b693-789734b64cdb\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " Jul 29 08:52:23.555717 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.555613 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-ui-tls\") pod \"7aad2c76-7155-450e-b693-789734b64cdb\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " Jul 29 08:52:23.555717 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.555691 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-offline-tls\") pod \"7aad2c76-7155-450e-b693-789734b64cdb\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " Jul 29 08:52:23.555880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.555720 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-online-tls\") pod \"7aad2c76-7155-450e-b693-789734b64cdb\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " Jul 29 08:52:23.555880 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.555755 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dh8d\" (UniqueName: \"kubernetes.io/projected/7aad2c76-7155-450e-b693-789734b64cdb-kube-api-access-8dh8d\") pod \"7aad2c76-7155-450e-b693-789734b64cdb\" (UID: \"7aad2c76-7155-450e-b693-789734b64cdb\") " Jul 29 08:52:23.556024 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.555997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aad2c76-7155-450e-b693-789734b64cdb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7aad2c76-7155-450e-b693-789734b64cdb" (UID: "7aad2c76-7155-450e-b693-789734b64cdb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:52:23.556096 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.556022 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aad2c76-7155-450e-b693-789734b64cdb-feast-data" (OuterVolumeSpecName: "feast-data") pod "7aad2c76-7155-450e-b693-789734b64cdb" (UID: "7aad2c76-7155-450e-b693-789734b64cdb"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:52:23.563297 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.563241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7aad2c76-7155-450e-b693-789734b64cdb" (UID: "7aad2c76-7155-450e-b693-789734b64cdb"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:52:23.563425 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.563302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7aad2c76-7155-450e-b693-789734b64cdb" (UID: "7aad2c76-7155-450e-b693-789734b64cdb"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:52:23.563425 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.563363 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-online-tls" (OuterVolumeSpecName: "online-tls") pod "7aad2c76-7155-450e-b693-789734b64cdb" (UID: "7aad2c76-7155-450e-b693-789734b64cdb"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:52:23.563425 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.563375 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aad2c76-7155-450e-b693-789734b64cdb-kube-api-access-8dh8d" (OuterVolumeSpecName: "kube-api-access-8dh8d") pod "7aad2c76-7155-450e-b693-789734b64cdb" (UID: "7aad2c76-7155-450e-b693-789734b64cdb"). InnerVolumeSpecName "kube-api-access-8dh8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:52:23.643798 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.643775 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:52:23.657217 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.657197 2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7aad2c76-7155-450e-b693-789734b64cdb-feast-data\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.657291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.657219 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/7aad2c76-7155-450e-b693-789734b64cdb-registry-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.657291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.657233 2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-ui-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.657291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.657241 2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-offline-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.657291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.657249 2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7aad2c76-7155-450e-b693-789734b64cdb-online-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.657291 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.657256 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dh8d\" (UniqueName: \"kubernetes.io/projected/7aad2c76-7155-450e-b693-789734b64cdb-kube-api-access-8dh8d\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.757673 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.757647 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls\") pod \"1b59755b-1316-4779-be8a-98e93afe64ae\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " Jul 29 08:52:23.757764 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.757686 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls\") pod \"1b59755b-1316-4779-be8a-98e93afe64ae\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " Jul 29 08:52:23.757764 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.757707 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-ui-tls\") pod \"1b59755b-1316-4779-be8a-98e93afe64ae\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " Jul 29 08:52:23.757764 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.757760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jvbh\" (UniqueName: \"kubernetes.io/projected/1b59755b-1316-4779-be8a-98e93afe64ae-kube-api-access-8jvbh\") pod \"1b59755b-1316-4779-be8a-98e93afe64ae\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " Jul 29 08:52:23.757934 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.757788 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls\") pod \"1b59755b-1316-4779-be8a-98e93afe64ae\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " Jul 29 08:52:23.757934 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.757826 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b59755b-1316-4779-be8a-98e93afe64ae-feast-data\") pod \"1b59755b-1316-4779-be8a-98e93afe64ae\" (UID: \"1b59755b-1316-4779-be8a-98e93afe64ae\") " Jul 29 08:52:23.758382 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.758338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b59755b-1316-4779-be8a-98e93afe64ae-feast-data" (OuterVolumeSpecName: "feast-data") pod "1b59755b-1316-4779-be8a-98e93afe64ae" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:52:23.759812 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.759790 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1b59755b-1316-4779-be8a-98e93afe64ae" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:52:23.760208 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.760190 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls" (OuterVolumeSpecName: "online-tls") pod "1b59755b-1316-4779-be8a-98e93afe64ae" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:52:23.760260 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.760203 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1b59755b-1316-4779-be8a-98e93afe64ae" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:52:23.760390 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.760369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1b59755b-1316-4779-be8a-98e93afe64ae" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:52:23.760390 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.760376 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b59755b-1316-4779-be8a-98e93afe64ae-kube-api-access-8jvbh" (OuterVolumeSpecName: "kube-api-access-8jvbh") pod "1b59755b-1316-4779-be8a-98e93afe64ae" (UID: "1b59755b-1316-4779-be8a-98e93afe64ae"). InnerVolumeSpecName "kube-api-access-8jvbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:52:23.859206 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.859185 2580 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b59755b-1316-4779-be8a-98e93afe64ae-feast-data\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.859206 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.859204 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-registry-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.859321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.859212 2580 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-online-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.859321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.859221 2580 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-ui-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.859321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.859230 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvbh\" (UniqueName: \"kubernetes.io/projected/1b59755b-1316-4779-be8a-98e93afe64ae-kube-api-access-8jvbh\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:23.859321 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:23.859240 2580 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b59755b-1316-4779-be8a-98e93afe64ae-offline-tls\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:52:24.464565 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.464537 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" Jul 29 08:52:24.465097 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.464556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt" event={"ID":"7aad2c76-7155-450e-b693-789734b64cdb","Type":"ContainerDied","Data":"077499a94d3f440b614cdd3d22ddcb027a99b2ef731813516342c346609c5dfb"} Jul 29 08:52:24.465097 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.464611 2580 scope.go:117] "RemoveContainer" containerID="4f7d7a1996a5f8983273a45706d77bd34b0396d5197aba844428119ba2bbd03d" Jul 29 08:52:24.467727 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.467706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" event={"ID":"1b59755b-1316-4779-be8a-98e93afe64ae","Type":"ContainerDied","Data":"b0bcc4a5497d8143f35c463f2a987449a21b0bf327c06f2a2ffae07e54e5aa5a"} Jul 29 08:52:24.467829 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.467765 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q" Jul 29 08:52:24.473673 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.473650 2580 scope.go:117] "RemoveContainer" containerID="1a067435baf8eaf62988d190785b70c97df3a34060a868286ec76e720adc8047" Jul 29 08:52:24.480752 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.480737 2580 scope.go:117] "RemoveContainer" containerID="a70dd1d1b3f879764ab5b0590122ca9dff523193e6c9997748c8cdc256a978ae" Jul 29 08:52:24.487222 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.487198 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q"] Jul 29 08:52:24.488299 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.488285 2580 scope.go:117] "RemoveContainer" containerID="de25f495194d90a6cde11475b1b8bf7890915a891dd093b68d0fdbc6b0f01b08" Jul 29 08:52:24.492015 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.491995 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-777d6cc998-bsf5q"] Jul 29 08:52:24.495777 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.495760 2580 scope.go:117] "RemoveContainer" containerID="196c57b658ba265583beab0664da79a66c66341234870e9e6e683ec9c29dce52" Jul 29 08:52:24.502735 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.502717 2580 scope.go:117] "RemoveContainer" containerID="f64b53fb1310b69ea419e5a3eb719e1cb5655111d5e9e0e464188d4cb2941aae" Jul 29 08:52:24.503078 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.503063 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt"] Jul 29 08:52:24.508299 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.508279 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-db6955c6c-rg5nt"] Jul 29 08:52:24.510302 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.510286 2580 scope.go:117] "RemoveContainer" containerID="a3d7ffacace613a21772e2fa4e24c89a98e0cc2a758d6384d8a68a7921befc39" Jul 29 08:52:24.516819 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.516804 2580 scope.go:117] "RemoveContainer" containerID="edde2aa0a0b05b87a532600e058e92df2a4f63440efcb7a4e9c6380cc2c0c138" Jul 29 08:52:24.523413 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.523398 2580 scope.go:117] "RemoveContainer" containerID="18139062fe99ed490704cd56336c84d6b4afc3ec0bf5c01a014fe68aa10efd9d" Jul 29 08:52:24.529998 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.529983 2580 scope.go:117] "RemoveContainer" containerID="6792f6cdf3ae5c7d1f535bec1e7b3013a19dd7a03068a93ecf2488375a0d6c2a" Jul 29 08:52:24.536258 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:24.536244 2580 scope.go:117] "RemoveContainer" containerID="6f01725ea38243baee81617557fb66f95d49497ec79d0bd083305863a0a44ab5" Jul 29 08:52:25.901213 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:25.901184 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" path="/var/lib/kubelet/pods/1b59755b-1316-4779-be8a-98e93afe64ae/volumes" Jul 29 08:52:25.901788 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:25.901773 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aad2c76-7155-450e-b693-789734b64cdb" path="/var/lib/kubelet/pods/7aad2c76-7155-450e-b693-789734b64cdb/volumes" Jul 29 08:52:42.703322 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703241 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bgpwg/must-gather-crkxc"] Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703551 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="feast-apply" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703563 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="feast-apply" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703592 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="feast-apply" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703598 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="feast-apply" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703609 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="feast-init" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703614 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="feast-init" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703621 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703626 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703634 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="registry" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703639 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="registry" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703644 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="offline" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703649 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="offline" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703654 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703658 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703668 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703672 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703678 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="feast-init" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703684 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="feast-init" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703694 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703702 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703715 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="offline" Jul 29 08:52:42.703757 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703723 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="offline" Jul 29 08:52:42.704501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703782 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="offline" Jul 29 08:52:42.704501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703790 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="online" Jul 29 08:52:42.704501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703796 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="offline" Jul 29 08:52:42.704501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703802 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="registry" Jul 29 08:52:42.704501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703807 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="ui" Jul 29 08:52:42.704501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703812 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b59755b-1316-4779-be8a-98e93afe64ae" containerName="ui" Jul 29 08:52:42.704501 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.703817 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aad2c76-7155-450e-b693-789734b64cdb" containerName="online" Jul 29 08:52:42.706632 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.706617 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:42.709302 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.709274 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bgpwg\"/\"openshift-service-ca.crt\"" Jul 29 08:52:42.710371 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.710348 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bgpwg\"/\"kube-root-ca.crt\"" Jul 29 08:52:42.710511 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.710423 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bgpwg\"/\"default-dockercfg-gwcnc\"" Jul 29 08:52:42.714469 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.714427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgpwg/must-gather-crkxc"] Jul 29 08:52:42.793397 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.793359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-must-gather-output\") pod \"must-gather-crkxc\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:42.793397 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.793403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8jk\" (UniqueName: \"kubernetes.io/projected/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-kube-api-access-br8jk\") pod \"must-gather-crkxc\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:42.893993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.893946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-must-gather-output\") pod \"must-gather-crkxc\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:42.893993 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.893996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8jk\" (UniqueName: \"kubernetes.io/projected/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-kube-api-access-br8jk\") pod \"must-gather-crkxc\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:42.894274 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.894253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-must-gather-output\") pod \"must-gather-crkxc\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:42.901998 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:42.901969 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8jk\" (UniqueName: \"kubernetes.io/projected/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-kube-api-access-br8jk\") pod \"must-gather-crkxc\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:43.016403 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:43.016314 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:52:43.144344 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:43.144312 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgpwg/must-gather-crkxc"] Jul 29 08:52:43.148460 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:52:43.148404 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4b521f5_6a41_4535_a4c3_c8e2b44f035f.slice/crio-eab8d812087b66cf1212db00ba1c09e1c54c6f758582b120c33807bfd45659df WatchSource:0}: Error finding container eab8d812087b66cf1212db00ba1c09e1c54c6f758582b120c33807bfd45659df: Status 404 returned error can't find the container with id eab8d812087b66cf1212db00ba1c09e1c54c6f758582b120c33807bfd45659df Jul 29 08:52:43.150444 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:43.150411 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:52:43.533241 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:43.533207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgpwg/must-gather-crkxc" event={"ID":"b4b521f5-6a41-4535-a4c3-c8e2b44f035f","Type":"ContainerStarted","Data":"eab8d812087b66cf1212db00ba1c09e1c54c6f758582b120c33807bfd45659df"} Jul 29 08:52:49.562490 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:49.562450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgpwg/must-gather-crkxc" event={"ID":"b4b521f5-6a41-4535-a4c3-c8e2b44f035f","Type":"ContainerStarted","Data":"066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc"} Jul 29 08:52:49.562901 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:49.562505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgpwg/must-gather-crkxc" event={"ID":"b4b521f5-6a41-4535-a4c3-c8e2b44f035f","Type":"ContainerStarted","Data":"6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e"} Jul 29 08:52:49.580371 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:49.580305 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bgpwg/must-gather-crkxc" podStartSLOduration=1.621758179 podStartE2EDuration="7.580285069s" totalImagesPullingTime="5.95852689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:52:49.579150722 +0000 UTC m=+1904.158832404" watchObservedRunningTime="2026-07-29 08:52:49.580285069 +0000 UTC m=+1904.159966752" Jul 29 08:52:58.592586 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:58.592549 2580 generic.go:356] "Generic (PLEG): container finished" podID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerID="6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e" exitCode=0 Jul 29 08:52:58.592959 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:58.592590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgpwg/must-gather-crkxc" event={"ID":"b4b521f5-6a41-4535-a4c3-c8e2b44f035f","Type":"ContainerDied","Data":"6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e"} Jul 29 08:52:58.592959 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:58.592847 2580 scope.go:117] "RemoveContainer" containerID="6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e" Jul 29 08:52:59.457445 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:52:59.457412 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bgpwg_must-gather-crkxc_b4b521f5-6a41-4535-a4c3-c8e2b44f035f/gather/0.log" Jul 29 08:53:02.652968 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:02.652942 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fztml_62056413-e3c1-4562-b6ae-6ee8a49df104/global-pull-secret-syncer/0.log" Jul 29 08:53:02.813406 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:02.813377 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ssfvp_c7e3136e-5a0b-4c82-a3b4-5d43b35d442e/konnectivity-agent/0.log" Jul 29 08:53:02.882196 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:02.882160 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-229.ec2.internal_39e627514db70702d43d1afa7e7236cb/haproxy/0.log" Jul 29 08:53:04.789495 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:04.789460 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bgpwg/must-gather-crkxc"] Jul 29 08:53:04.789898 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:04.789612 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-bgpwg/must-gather-crkxc" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerName="copy" containerID="cri-o://066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc" gracePeriod=2 Jul 29 08:53:04.791593 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:04.791572 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bgpwg/must-gather-crkxc"] Jul 29 08:53:04.791698 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:53:04.791592 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crkxc\" is forbidden: User \"system:node:ip-10-0-138-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bgpwg\": no relationship found between node 'ip-10-0-138-229.ec2.internal' and this object" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:53:05.013611 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.013590 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bgpwg_must-gather-crkxc_b4b521f5-6a41-4535-a4c3-c8e2b44f035f/copy/0.log" Jul 29 08:53:05.014944 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.014928 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:53:05.017001 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:53:05.016977 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crkxc\" is forbidden: User \"system:node:ip-10-0-138-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bgpwg\": no relationship found between node 'ip-10-0-138-229.ec2.internal' and this object" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:53:05.152076 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.152057 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br8jk\" (UniqueName: \"kubernetes.io/projected/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-kube-api-access-br8jk\") pod \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " Jul 29 08:53:05.152184 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.152117 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-must-gather-output\") pod \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\" (UID: \"b4b521f5-6a41-4535-a4c3-c8e2b44f035f\") " Jul 29 08:53:05.152473 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.152448 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "b4b521f5-6a41-4535-a4c3-c8e2b44f035f" (UID: "b4b521f5-6a41-4535-a4c3-c8e2b44f035f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:53:05.154104 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.154085 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-kube-api-access-br8jk" (OuterVolumeSpecName: "kube-api-access-br8jk") pod "b4b521f5-6a41-4535-a4c3-c8e2b44f035f" (UID: "b4b521f5-6a41-4535-a4c3-c8e2b44f035f"). InnerVolumeSpecName "kube-api-access-br8jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:53:05.252843 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.252820 2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-must-gather-output\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:53:05.252843 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.252841 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br8jk\" (UniqueName: \"kubernetes.io/projected/b4b521f5-6a41-4535-a4c3-c8e2b44f035f-kube-api-access-br8jk\") on node \"ip-10-0-138-229.ec2.internal\" DevicePath \"\"" Jul 29 08:53:05.616404 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.616359 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bgpwg_must-gather-crkxc_b4b521f5-6a41-4535-a4c3-c8e2b44f035f/copy/0.log" Jul 29 08:53:05.616671 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.616650 2580 generic.go:356] "Generic (PLEG): container finished" podID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerID="066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc" exitCode=143 Jul 29 08:53:05.616752 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.616692 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:53:05.616752 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.616704 2580 scope.go:117] "RemoveContainer" containerID="066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc" Jul 29 08:53:05.619206 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:53:05.619174 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crkxc\" is forbidden: User \"system:node:ip-10-0-138-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bgpwg\": no relationship found between node 'ip-10-0-138-229.ec2.internal' and this object" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:53:05.623972 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.623919 2580 scope.go:117] "RemoveContainer" containerID="6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e" Jul 29 08:53:05.626236 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:53:05.626206 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crkxc\" is forbidden: User \"system:node:ip-10-0-138-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bgpwg\": no relationship found between node 'ip-10-0-138-229.ec2.internal' and this object" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:53:05.634711 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.634692 2580 scope.go:117] "RemoveContainer" containerID="066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc" Jul 29 08:53:05.634950 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:53:05.634931 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc\": container with ID starting with 066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc not found: ID does not exist" containerID="066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc" Jul 29 08:53:05.634996 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.634958 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc"} err="failed to get container status \"066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc\": rpc error: code = NotFound desc = could not find container \"066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc\": container with ID starting with 066772c2bef0ddbf484600b78bb1e38846e0ae54eaf0edac63fd2ff7643e91cc not found: ID does not exist" Jul 29 08:53:05.634996 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.634975 2580 scope.go:117] "RemoveContainer" containerID="6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e" Jul 29 08:53:05.635180 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:53:05.635161 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e\": container with ID starting with 6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e not found: ID does not exist" containerID="6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e" Jul 29 08:53:05.635218 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.635188 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e"} err="failed to get container status \"6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e\": rpc error: code = NotFound desc = could not find container \"6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e\": container with ID starting with 6303becc1dd134caf24f5f9a8a0fdaf089a7989d3693d5e42abf72282c95af8e not found: ID does not exist" Jul 29 08:53:05.899058 ip-10-0-138-229 kubenswrapper[2580]: E0729 08:53:05.898988 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-crkxc\" is forbidden: User \"system:node:ip-10-0-138-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bgpwg\": no relationship found between node 'ip-10-0-138-229.ec2.internal' and this object" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" pod="openshift-must-gather-bgpwg/must-gather-crkxc" Jul 29 08:53:05.899735 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:05.899717 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" path="/var/lib/kubelet/pods/b4b521f5-6a41-4535-a4c3-c8e2b44f035f/volumes" Jul 29 08:53:07.292447 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:07.292399 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g25xf_a3e30b65-ffd5-47fd-bfcd-f33b6848a76b/node-exporter/0.log" Jul 29 08:53:07.318956 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:07.318930 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g25xf_a3e30b65-ffd5-47fd-bfcd-f33b6848a76b/kube-rbac-proxy/0.log" Jul 29 08:53:07.349820 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:07.349800 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g25xf_a3e30b65-ffd5-47fd-bfcd-f33b6848a76b/init-textfile/0.log" Jul 29 08:53:08.223452 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:08.223360 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6d9c9ddd-fdk42_6365423e-fc6c-48d9-b64f-68e18264b2c1/telemeter-client/0.log" Jul 29 08:53:08.245531 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:08.245503 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6d9c9ddd-fdk42_6365423e-fc6c-48d9-b64f-68e18264b2c1/reload/0.log" Jul 29 08:53:08.270947 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:08.270923 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b6d9c9ddd-fdk42_6365423e-fc6c-48d9-b64f-68e18264b2c1/kube-rbac-proxy/0.log" Jul 29 08:53:09.461084 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.461063 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-gwk8t_18084e73-8406-4101-ac8e-7cb9325be37a/networking-console-plugin/0.log" Jul 29 08:53:09.810739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.808818 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l"] Jul 29 08:53:09.810739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.809357 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerName="copy" Jul 29 08:53:09.810739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.809373 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerName="copy" Jul 29 08:53:09.810739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.809405 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerName="gather" Jul 29 08:53:09.810739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.809414 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerName="gather" Jul 29 08:53:09.810739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.809540 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerName="gather" Jul 29 08:53:09.810739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.809565 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b521f5-6a41-4535-a4c3-c8e2b44f035f" containerName="copy" Jul 29 08:53:09.812138 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.812112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.814540 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.814519 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qrb25\"/\"openshift-service-ca.crt\"" Jul 29 08:53:09.814702 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.814685 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qrb25\"/\"kube-root-ca.crt\"" Jul 29 08:53:09.815626 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.815611 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qrb25\"/\"default-dockercfg-p4qm6\"" Jul 29 08:53:09.819545 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.819526 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l"] Jul 29 08:53:09.881614 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.881589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwn7\" (UniqueName: \"kubernetes.io/projected/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-kube-api-access-wlwn7\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.881732 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.881617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-sys\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.881732 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.881643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-lib-modules\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.881841 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.881743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-proc\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.881841 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.881779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-podres\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.982997 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.982964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-proc\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.982997 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.982998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-podres\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.983169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.983021 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwn7\" (UniqueName: \"kubernetes.io/projected/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-kube-api-access-wlwn7\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.983169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.983038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-sys\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.983169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.983074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-lib-modules\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.983169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.983078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-proc\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.983169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.983114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-sys\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.983169 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.983153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-podres\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.983360 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.983207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-lib-modules\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:09.990646 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:09.990626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwn7\" (UniqueName: \"kubernetes.io/projected/7c49127a-77ca-4f7e-93e8-bb44ad5fdca9-kube-api-access-wlwn7\") pod \"perf-node-gather-daemonset-qmv7l\" (UID: \"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9\") " pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:10.122953 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:10.122907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:10.245371 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:10.245265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l"] Jul 29 08:53:10.248053 ip-10-0-138-229 kubenswrapper[2580]: W0729 08:53:10.248027 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c49127a_77ca_4f7e_93e8_bb44ad5fdca9.slice/crio-0df85f6bb48f1f302d6664d517b0f24ffb0a7527e6477eb472c4d12dfd51e226 WatchSource:0}: Error finding container 0df85f6bb48f1f302d6664d517b0f24ffb0a7527e6477eb472c4d12dfd51e226: Status 404 returned error can't find the container with id 0df85f6bb48f1f302d6664d517b0f24ffb0a7527e6477eb472c4d12dfd51e226 Jul 29 08:53:10.633721 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:10.633687 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" event={"ID":"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9","Type":"ContainerStarted","Data":"82c9d3493f63170d8f41c0508bb507d38207733769626cdb178b8e7558e06997"} Jul 29 08:53:10.634073 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:10.633741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" event={"ID":"7c49127a-77ca-4f7e-93e8-bb44ad5fdca9","Type":"ContainerStarted","Data":"0df85f6bb48f1f302d6664d517b0f24ffb0a7527e6477eb472c4d12dfd51e226"} Jul 29 08:53:10.634073 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:10.633792 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:10.650418 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:10.650357 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" podStartSLOduration=1.650337403 podStartE2EDuration="1.650337403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:53:10.650000928 +0000 UTC m=+1925.229682605" watchObservedRunningTime="2026-07-29 08:53:10.650337403 +0000 UTC m=+1925.230019067" Jul 29 08:53:11.524752 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:11.524720 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-khm2f_5121a769-ab74-4c2e-8f1e-d7c09c88d44f/dns/0.log" Jul 29 08:53:11.544547 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:11.544506 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-khm2f_5121a769-ab74-4c2e-8f1e-d7c09c88d44f/kube-rbac-proxy/0.log" Jul 29 08:53:11.630438 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:11.630416 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lw44r_5beacbc7-a61e-4621-912e-d3807d189d2d/dns-node-resolver/0.log" Jul 29 08:53:12.134750 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:12.134728 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w5j4b_0006c3a5-faa2-4b14-a57c-1e1b468041a8/node-ca/0.log" Jul 29 08:53:13.226397 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:13.226367 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9j5xf_7b6c47ce-1287-4bef-80d5-42b69b8c5431/serve-healthcheck-canary/0.log" Jul 29 08:53:13.658110 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:13.658078 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zbsm_ff71c6ab-1aa9-4620-a002-c09e43bd308f/kube-rbac-proxy/0.log" Jul 29 08:53:13.677349 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:13.677323 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zbsm_ff71c6ab-1aa9-4620-a002-c09e43bd308f/exporter/0.log" Jul 29 08:53:13.697808 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:13.697761 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zbsm_ff71c6ab-1aa9-4620-a002-c09e43bd308f/extractor/0.log" Jul 29 08:53:16.646739 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:16.646711 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qrb25/perf-node-gather-daemonset-qmv7l" Jul 29 08:53:18.982759 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:18.982724 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p2w7k_5ebeefc4-a87d-46db-8f40-d2cebb70eca1/kube-multus-additional-cni-plugins/0.log" Jul 29 08:53:19.003840 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.003796 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p2w7k_5ebeefc4-a87d-46db-8f40-d2cebb70eca1/egress-router-binary-copy/0.log" Jul 29 08:53:19.027209 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.027169 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p2w7k_5ebeefc4-a87d-46db-8f40-d2cebb70eca1/cni-plugins/0.log" Jul 29 08:53:19.047361 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.047343 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p2w7k_5ebeefc4-a87d-46db-8f40-d2cebb70eca1/bond-cni-plugin/0.log" Jul 29 08:53:19.067372 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.067352 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p2w7k_5ebeefc4-a87d-46db-8f40-d2cebb70eca1/routeoverride-cni/0.log" Jul 29 08:53:19.089067 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.089049 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p2w7k_5ebeefc4-a87d-46db-8f40-d2cebb70eca1/whereabouts-cni-bincopy/0.log" Jul 29 08:53:19.111358 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.111302 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p2w7k_5ebeefc4-a87d-46db-8f40-d2cebb70eca1/whereabouts-cni/0.log" Jul 29 08:53:19.496079 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.496053 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-svvdf_928e848d-cc8f-4abe-b018-ea9084a8f303/kube-multus/0.log" Jul 29 08:53:19.635254 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.635229 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vmj9l_bf109907-5eb4-404b-8454-e87a1e2c5849/network-metrics-daemon/0.log" Jul 29 08:53:19.657220 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:19.657202 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vmj9l_bf109907-5eb4-404b-8454-e87a1e2c5849/kube-rbac-proxy/0.log" Jul 29 08:53:20.459731 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.459706 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/ovn-controller/0.log" Jul 29 08:53:20.477142 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.477110 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/ovn-acl-logging/0.log" Jul 29 08:53:20.503373 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.503354 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/kube-rbac-proxy-node/0.log" Jul 29 08:53:20.527461 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.527421 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/kube-rbac-proxy-ovn-metrics/0.log" Jul 29 08:53:20.550664 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.550592 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/northd/0.log" Jul 29 08:53:20.569654 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.569622 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/nbdb/0.log" Jul 29 08:53:20.593309 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.593293 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/sbdb/0.log" Jul 29 08:53:20.653155 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:20.653125 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ffbdj_222f904f-3dff-40ac-a735-40260879417f/ovnkube-controller/0.log" Jul 29 08:53:22.134778 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:22.134745 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-hqzk8_a5014e06-bb11-4b83-854d-74f033f51cb9/network-check-target-container/0.log" Jul 29 08:53:23.044731 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:23.044701 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-pb88v_13fd06f4-1b82-4049-b980-0ef9d1ae0742/iptables-alerter/0.log" Jul 29 08:53:23.707140 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:23.707113 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-m7c5r_40efe082-36d8-470a-9c42-754292673df6/tuned/0.log" Jul 29 08:53:26.261088 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:26.261054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-5d6c75f84f-fcsxb_00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a/service-ca-operator/1.log" Jul 29 08:53:26.262801 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:26.262777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-5d6c75f84f-fcsxb_00d457a6-d3fe-4cf0-92e0-66c6e92bbe9a/service-ca-operator/0.log" Jul 29 08:53:26.582530 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:26.582470 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-658d7b7c78-m2rgk_78244da5-7fdd-4da8-920f-67d56984eaee/service-ca-controller/0.log" Jul 29 08:53:27.100381 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:27.100345 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-s6mxr_3be7fa5e-32eb-4151-9f4a-9b80f2420d72/csi-driver/0.log" Jul 29 08:53:27.121032 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:27.121010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-s6mxr_3be7fa5e-32eb-4151-9f4a-9b80f2420d72/csi-node-driver-registrar/0.log" Jul 29 08:53:27.140722 ip-10-0-138-229 kubenswrapper[2580]: I0729 08:53:27.140696 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-s6mxr_3be7fa5e-32eb-4151-9f4a-9b80f2420d72/csi-liveness-probe/0.log"