Sep 13 19:58:23.808467 ip-10-0-141-146 systemd[1]: Starting Kubernetes Kubelet... Sep 13 19:58:24.225561 ip-10-0-141-146 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 13 19:58:24.225561 ip-10-0-141-146 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 13 19:58:24.225561 ip-10-0-141-146 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 13 19:58:24.225561 ip-10-0-141-146 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 13 19:58:24.225561 ip-10-0-141-146 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 13 19:58:24.227944 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.227859 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 13 19:58:24.232040 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232025 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 13 19:58:24.232040 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232041 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232044 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232048 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232051 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232054 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232057 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232060 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232071 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232074 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232077 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232080 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232083 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232085 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232088 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232090 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232094 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232096 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232099 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232102 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232105 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 13 19:58:24.232104 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232108 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232110 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232113 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232116 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232119 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232122 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232124 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232128 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232130 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232133 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232136 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232138 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232141 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232143 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232146 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232148 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232151 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232153 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232156 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232159 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 13 19:58:24.232588 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232161 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232164 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232166 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232169 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232171 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232173 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232178 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232193 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232196 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232199 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232201 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232203 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232206 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232208 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232213 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232217 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232221 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232225 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232228 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 13 19:58:24.233119 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232231 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232234 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232236 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232239 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232242 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232244 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232247 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232250 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232252 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232255 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232257 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232260 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232265 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232267 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232270 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232272 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232274 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232277 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232280 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232282 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 13 19:58:24.233594 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232285 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232287 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232290 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232293 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232295 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232299 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232302 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232306 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232309 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232312 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232314 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232317 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232319 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232322 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232324 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232327 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232329 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232332 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 13 19:58:24.234069 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.232334 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 13 19:58:24.236070 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236045 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 13 19:58:24.236070 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236068 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 13 19:58:24.236070 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236075 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236080 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236085 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236090 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236097 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236109 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236113 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236118 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236123 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236127 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236132 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236136 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236141 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236145 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236150 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236154 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236158 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236167 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236172 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236176 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 13 19:58:24.236278 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236196 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236201 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236205 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236209 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236214 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236218 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236223 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236228 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236232 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236242 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236247 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236251 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236255 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236260 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236268 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236272 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236277 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236282 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236286 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 13 19:58:24.236917 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236290 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236295 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236300 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236309 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236314 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236318 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236322 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236327 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236331 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236335 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236340 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236344 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236348 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236353 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236357 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236367 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236371 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236375 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236382 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 13 19:58:24.237441 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236388 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236392 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236398 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236405 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236410 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236414 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236418 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236432 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236439 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236444 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236448 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236452 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236457 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236462 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236466 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236470 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236475 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236479 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236484 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 13 19:58:24.237913 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236488 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236496 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236499 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236501 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236504 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236507 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236510 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236513 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236515 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236518 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236521 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236524 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236526 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236529 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236534 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236537 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236541 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236545 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236550 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.236554 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237138 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 13 19:58:24.238397 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237145 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237148 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237151 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237153 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237157 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237159 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237162 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237165 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237168 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237171 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237174 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237178 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237195 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237198 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237202 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237204 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237207 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237209 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237212 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237214 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 13 19:58:24.238908 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237217 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237220 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237223 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237227 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237229 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237232 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237235 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237237 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237240 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237243 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237245 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237248 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237251 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237253 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237256 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237259 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237261 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237264 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237266 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 13 19:58:24.239399 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237269 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237272 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237275 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237278 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237280 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237283 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237285 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237288 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237290 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237294 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237296 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237299 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237301 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237304 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237306 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237309 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237311 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237316 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237319 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 13 19:58:24.239865 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237323 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237327 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237330 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237332 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237335 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237338 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237341 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237344 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237347 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237351 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237354 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237357 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237360 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237362 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237368 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237370 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237373 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237376 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237379 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237381 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 13 19:58:24.240341 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237384 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237386 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237389 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237392 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237395 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237397 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237400 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237402 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237405 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237408 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237411 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237413 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237416 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237419 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237421 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237424 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237426 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237429 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237432 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.237434 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237505 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 13 19:58:24.240832 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237540 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237546 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237551 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237555 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237558 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237564 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237568 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237572 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237575 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237578 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237581 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237584 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237587 2578 flags.go:64] FLAG: --cgroup-root="" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237590 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237593 2578 flags.go:64] FLAG: --client-ca-file="" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237597 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237600 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237605 2578 flags.go:64] FLAG: --cluster-domain="" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237608 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237611 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237614 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237617 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237621 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 13 19:58:24.241348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237637 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237641 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237645 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237649 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237651 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237656 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237659 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237662 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237666 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237670 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237673 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237676 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237679 2578 flags.go:64] FLAG: --enable-server="true" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237682 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237687 2578 flags.go:64] FLAG: --event-burst="100" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237690 2578 flags.go:64] FLAG: --event-qps="50" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237693 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237696 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237699 2578 flags.go:64] FLAG: --eviction-hard="" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237703 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237706 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237709 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237712 2578 flags.go:64] FLAG: --eviction-soft="" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237716 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237719 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 13 19:58:24.241897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237722 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237725 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237728 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237731 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237734 2578 flags.go:64] FLAG: --feature-gates="" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237738 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237741 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237744 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237748 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237751 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237754 2578 flags.go:64] FLAG: --help="false" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237757 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237762 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237765 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237768 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237772 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237775 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237778 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237781 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237784 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237787 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237791 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237794 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237797 2578 flags.go:64] FLAG: --kube-reserved="" Sep 13 19:58:24.242660 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237800 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237803 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237806 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237809 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237812 2578 flags.go:64] FLAG: --lock-file="" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237815 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237818 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237821 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237826 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237829 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237832 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237835 2578 flags.go:64] FLAG: --logging-format="text" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237838 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237841 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237844 2578 flags.go:64] FLAG: --manifest-url="" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237847 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237851 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237854 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237859 2578 flags.go:64] FLAG: --max-pods="110" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237862 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237866 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237869 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237872 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237875 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237877 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 13 19:58:24.243330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237880 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237888 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237891 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237894 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237898 2578 flags.go:64] FLAG: --pod-cidr="" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237901 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237907 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237910 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237913 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237916 2578 flags.go:64] FLAG: --port="10250" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237919 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237922 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ee0784f5725180c9" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237926 2578 flags.go:64] FLAG: --qos-reserved="" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237929 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237932 2578 flags.go:64] FLAG: --register-node="true" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237935 2578 flags.go:64] FLAG: --register-with-taints="" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237938 2578 flags.go:64] FLAG: --registry-burst="10" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237942 2578 flags.go:64] FLAG: --registry-qps="5" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237945 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237948 2578 flags.go:64] FLAG: --reserved-memory="" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237951 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237954 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237957 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237960 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237963 2578 flags.go:64] FLAG: --runonce="false" Sep 13 19:58:24.243974 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237966 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237969 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237972 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237976 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237979 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237982 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237984 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237987 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237990 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237993 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237996 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.237999 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238002 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238005 2578 flags.go:64] FLAG: --system-cgroups="" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238008 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238014 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238017 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238020 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238025 2578 flags.go:64] FLAG: --tls-min-version="" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238028 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238030 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238033 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238036 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238039 2578 flags.go:64] FLAG: --v="2" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238043 2578 flags.go:64] FLAG: --version="false" Sep 13 19:58:24.244604 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238047 2578 flags.go:64] FLAG: --vmodule="" Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238051 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238054 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238169 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238173 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238176 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238179 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238197 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238202 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238206 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238212 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238215 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238218 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238222 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238226 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238229 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238231 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238234 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238238 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 13 19:58:24.245274 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238241 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238243 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238246 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238249 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238251 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238254 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238257 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238260 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238262 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238265 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238273 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238276 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238278 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238281 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238284 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238287 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238289 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238292 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238295 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238297 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 13 19:58:24.245749 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238300 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238302 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238305 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238309 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238312 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238315 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238318 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238320 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238323 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238326 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238328 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238331 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238334 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238336 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238339 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238342 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238345 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238348 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238350 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238353 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238355 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 13 19:58:24.246264 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238358 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238360 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238363 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238365 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238368 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238371 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238373 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238376 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238380 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238384 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238387 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238390 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238393 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238395 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238399 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238402 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238405 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238408 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238410 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 13 19:58:24.246824 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238413 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238416 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238418 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238421 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238424 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238426 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238429 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238432 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238435 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238437 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238440 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238442 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238445 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238447 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238450 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238453 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238455 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238458 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238461 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 13 19:58:24.247322 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238463 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238467 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238470 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.238472 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.238478 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.246474 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.246490 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.247411 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.247420 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247475 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247480 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247485 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247488 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247491 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247494 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 13 19:58:24.247820 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247497 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247500 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247503 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247505 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247508 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247511 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247513 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247516 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247519 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247521 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247524 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247526 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247529 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247532 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247534 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247537 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247540 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247542 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247545 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247548 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 13 19:58:24.248215 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247550 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247553 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247556 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247560 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247562 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247565 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247568 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247570 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247573 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247576 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247578 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247581 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247583 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247586 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247589 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247591 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247594 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247596 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247599 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247602 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 13 19:58:24.248716 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247604 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247607 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247609 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247612 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247614 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247617 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247620 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247622 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247624 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247627 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247630 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247632 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247635 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247637 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247641 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247644 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247647 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247651 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247655 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247658 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 13 19:58:24.249208 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247661 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247664 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247667 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247669 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247672 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247674 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247677 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247679 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247682 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247684 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247688 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247692 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247695 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247720 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247724 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247728 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247731 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247734 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247737 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 13 19:58:24.249694 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247739 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247742 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247745 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247748 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247752 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247755 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247758 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247762 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247764 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247767 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247770 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247773 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247776 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247779 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 13 19:58:24.250204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.247785 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247894 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247899 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247902 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247906 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247909 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247912 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247915 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247917 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247921 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247925 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247928 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247931 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247934 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247936 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247939 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247941 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247944 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247947 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 13 19:58:24.250558 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247949 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247952 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247954 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247957 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247959 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247963 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247965 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247968 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247971 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247973 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247977 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247980 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247982 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247984 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247987 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247990 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247992 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247995 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.247997 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248000 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 13 19:58:24.251074 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248003 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248005 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248007 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248010 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248012 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248015 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248017 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248020 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248023 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248025 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248027 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248030 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248032 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248035 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248039 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248042 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248045 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248048 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248051 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248053 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 13 19:58:24.251654 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248056 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248058 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248061 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248064 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248067 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248069 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248072 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248074 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248077 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248079 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248082 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248085 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248087 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248090 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248092 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248095 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248097 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248100 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248102 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 13 19:58:24.252279 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248105 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248107 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248110 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248113 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248115 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248118 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248120 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248123 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248126 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248130 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248133 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248135 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248138 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248141 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248143 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248145 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248148 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248151 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248153 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248156 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 13 19:58:24.252746 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248158 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 13 19:58:24.253452 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:24.248160 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 13 19:58:24.253452 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.248165 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 13 19:58:24.253452 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.248290 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 13 19:58:24.253452 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.250767 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 13 19:58:24.253452 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.251872 2578 server.go:1019] "Starting client certificate rotation" Sep 13 19:58:24.253452 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.251992 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 13 19:58:24.253452 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.252044 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 13 19:58:24.254738 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.254724 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 13 19:58:24.255853 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.255834 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 13 19:58:24.269010 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.268988 2578 log.go:25] "Validated CRI v1 runtime API" Sep 13 19:58:24.272345 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.272330 2578 log.go:25] "Validated CRI v1 image API" Sep 13 19:58:24.273113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.273094 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 13 19:58:24.275564 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.275546 2578 fs.go:135] Filesystem UUIDs: map[3b046e0d-bfc8-4c59-ad19-45e342a7e029:/dev/nvme0n1p4 5cb22556-5ba4-496f-90cd-b45f03915906:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 13 19:58:24.275613 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.275564 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 13 19:58:24.281105 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.281084 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 13 19:58:24.281769 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.281665 2578 manager.go:216] Machine: {Timestamp:2026-09-13 19:58:24.280281672 +0000 UTC m=+0.363426578 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104245 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec208c97e1b521b4550a839121d3d438 SystemUUID:ec208c97-e1b5-21b4-550a-839121d3d438 BootID:05fb02f5-20ed-46e6-bc2a-f91877e85340 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:42:7d:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:42:7d:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:e1:bf:87:55:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 13 19:58:24.281769 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.281764 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 13 19:58:24.281867 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.281842 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 13 19:58:24.285506 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.285484 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 13 19:58:24.285643 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.285509 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 13 19:58:24.287045 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.287035 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 13 19:58:24.287087 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.287048 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 13 19:58:24.287087 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.287055 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 13 19:58:24.287889 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.287878 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 13 19:58:24.289084 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.289074 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 13 19:58:24.293710 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.293699 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 13 19:58:24.293813 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.293804 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 13 19:58:24.293843 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.293832 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 13 19:58:24.293843 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.293841 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 13 19:58:24.293926 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.293859 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 13 19:58:24.293926 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.293868 2578 kubelet.go:398] "Adding apiserver pod source" Sep 13 19:58:24.293926 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.293876 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 13 19:58:24.294221 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.294210 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 13 19:58:24.294260 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.294229 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 13 19:58:24.296497 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.296480 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 13 19:58:24.299280 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.299266 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 13 19:58:24.299349 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.299296 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 13 19:58:24.300492 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.300474 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hhpmc" Sep 13 19:58:24.301432 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301418 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301436 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301442 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301448 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301458 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301472 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301488 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301493 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 13 19:58:24.301496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301500 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 13 19:58:24.301793 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301507 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 13 19:58:24.301793 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301520 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 13 19:58:24.301793 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.301526 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 13 19:58:24.302080 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.302059 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 13 19:58:24.302080 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.302063 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 13 19:58:24.303364 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.303352 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 13 19:58:24.303427 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.303367 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 13 19:58:24.306232 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.306209 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 13 19:58:24.307494 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.307475 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hhpmc" Sep 13 19:58:24.309489 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.309475 2578 server.go:1268] "Started kubelet" Sep 13 19:58:24.309582 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.309555 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 13 19:58:24.309687 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.309557 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 13 19:58:24.309789 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.309708 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 13 19:58:24.310241 ip-10-0-141-146 systemd[1]: Started Kubernetes Kubelet. Sep 13 19:58:24.311998 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.311980 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 13 19:58:24.312154 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.312141 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 13 19:58:24.314686 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.314665 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 13 19:58:24.317696 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.317675 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 13 19:58:24.318364 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.318345 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 13 19:58:24.318450 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.318433 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:24.319064 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319050 2578 factory.go:55] Registering systemd factory Sep 13 19:58:24.319064 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319065 2578 factory.go:223] Registration of the systemd container factory successfully Sep 13 19:58:24.319197 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319097 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 13 19:58:24.319197 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319101 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 13 19:58:24.319197 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319109 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 13 19:58:24.319197 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319156 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 13 19:58:24.319197 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319163 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 13 19:58:24.319355 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319285 2578 factory.go:153] Registering CRI-O factory Sep 13 19:58:24.319355 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319296 2578 factory.go:223] Registration of the crio container factory successfully Sep 13 19:58:24.319355 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319338 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 13 19:58:24.319355 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319351 2578 factory.go:103] Registering Raw factory Sep 13 19:58:24.319465 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319362 2578 manager.go:1196] Started watching for new ooms in manager Sep 13 19:58:24.319720 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.319702 2578 manager.go:349] Starting recovery of all containers Sep 13 19:58:24.325296 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.325115 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 13 19:58:24.328534 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.328503 2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-146.ec2.internal\" not found" node="ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.331959 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.331943 2578 manager.go:354] Recovery completed Sep 13 19:58:24.338737 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.338724 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 13 19:58:24.340369 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.340355 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientMemory" Sep 13 19:58:24.340426 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.340381 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasNoDiskPressure" Sep 13 19:58:24.340426 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.340391 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientPID" Sep 13 19:58:24.340874 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.340861 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 13 19:58:24.340938 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.340874 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 13 19:58:24.340938 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.340893 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 13 19:58:24.340938 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.340923 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 13 19:58:24.341686 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.341671 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientMemory" Sep 13 19:58:24.341750 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.341702 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasNoDiskPressure" Sep 13 19:58:24.341750 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.341717 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientPID" Sep 13 19:58:24.343066 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.343054 2578 policy_none.go:49] "None policy: Start" Sep 13 19:58:24.343110 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.343071 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 13 19:58:24.343110 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.343083 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 13 19:58:24.345053 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.345042 2578 policy_none.go:47] "Start" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.383779 2578 manager.go:337] "Starting Device Plugin manager" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.383828 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.383840 2578 server.go:87] "Starting device plugin registration server" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.384180 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.384211 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.384317 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.384399 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.384409 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.384925 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 13 19:58:24.394307 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.384973 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:24.412368 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.412342 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 13 19:58:24.413512 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.413493 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 13 19:58:24.413610 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.413536 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 13 19:58:24.413610 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.413557 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 13 19:58:24.413610 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.413596 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 13 19:58:24.415712 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.415686 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 13 19:58:24.485196 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.485119 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 13 19:58:24.486470 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.486454 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientMemory" Sep 13 19:58:24.486565 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.486480 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasNoDiskPressure" Sep 13 19:58:24.486565 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.486489 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientPID" Sep 13 19:58:24.486565 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.486514 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.494164 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.494150 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.494227 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.494170 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-146.ec2.internal\": node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:24.513713 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.513688 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal"] Sep 13 19:58:24.513796 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.513752 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 13 19:58:24.515016 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.515002 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientMemory" Sep 13 19:58:24.515096 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.515027 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasNoDiskPressure" Sep 13 19:58:24.515096 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.515044 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientPID" Sep 13 19:58:24.516761 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.516744 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:24.517411 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.517399 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 13 19:58:24.517497 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.517482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.517544 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.517514 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 13 19:58:24.518078 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.518064 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientMemory" Sep 13 19:58:24.518142 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.518088 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasNoDiskPressure" Sep 13 19:58:24.518142 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.518103 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientPID" Sep 13 19:58:24.518142 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.518127 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientMemory" Sep 13 19:58:24.518272 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.518145 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasNoDiskPressure" Sep 13 19:58:24.518272 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.518159 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientPID" Sep 13 19:58:24.519907 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.519888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/419cc3454d80e28154e06a9c276d2a69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal\" (UID: \"419cc3454d80e28154e06a9c276d2a69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.519991 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.519910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/419cc3454d80e28154e06a9c276d2a69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal\" (UID: \"419cc3454d80e28154e06a9c276d2a69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.519991 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.519927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c92652107ed38ad17b29af94e128de12-config\") pod \"kube-apiserver-proxy-ip-10-0-141-146.ec2.internal\" (UID: \"c92652107ed38ad17b29af94e128de12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.520428 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.520406 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.520428 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.520429 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 13 19:58:24.521036 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.521022 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientMemory" Sep 13 19:58:24.521100 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.521041 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasNoDiskPressure" Sep 13 19:58:24.521100 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.521050 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeHasSufficientPID" Sep 13 19:58:24.548725 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.548703 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-146.ec2.internal\" not found" node="ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.552950 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.552931 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-146.ec2.internal\" not found" node="ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.617339 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.617318 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:24.620575 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.620562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/419cc3454d80e28154e06a9c276d2a69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal\" (UID: \"419cc3454d80e28154e06a9c276d2a69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.620618 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.620583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c92652107ed38ad17b29af94e128de12-config\") pod \"kube-apiserver-proxy-ip-10-0-141-146.ec2.internal\" (UID: \"c92652107ed38ad17b29af94e128de12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.620618 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.620600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/419cc3454d80e28154e06a9c276d2a69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal\" (UID: \"419cc3454d80e28154e06a9c276d2a69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.620687 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.620640 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/419cc3454d80e28154e06a9c276d2a69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal\" (UID: \"419cc3454d80e28154e06a9c276d2a69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.620687 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.620645 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/419cc3454d80e28154e06a9c276d2a69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal\" (UID: \"419cc3454d80e28154e06a9c276d2a69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.620746 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.620683 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c92652107ed38ad17b29af94e128de12-config\") pod \"kube-apiserver-proxy-ip-10-0-141-146.ec2.internal\" (UID: \"c92652107ed38ad17b29af94e128de12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.718281 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.718234 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:24.818994 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.818919 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:24.851322 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.851297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.856526 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:24.856222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" Sep 13 19:58:24.919087 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:24.919048 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:25.025919 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:25.025891 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:25.126640 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:25.126571 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:25.227678 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:25.227646 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:25.251693 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.251662 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 13 19:58:25.251815 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.251797 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 13 19:58:25.251857 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.251803 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 13 19:58:25.313941 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.313870 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-12 19:53:24 +0000 UTC" deadline="2028-05-11 19:04:35.522949334 +0000 UTC" Sep 13 19:58:25.313941 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.313899 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14543h6m10.209053649s" Sep 13 19:58:25.327825 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:25.327804 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:25.427986 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:25.427925 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-146.ec2.internal\" not found" Sep 13 19:58:25.497253 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.497236 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 13 19:58:25.506583 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:25.506551 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419cc3454d80e28154e06a9c276d2a69.slice/crio-eace16a810da0b586127d067b7b36e79ed0c30bb897ea139e698ac9cd15fa306 WatchSource:0}: Error finding container eace16a810da0b586127d067b7b36e79ed0c30bb897ea139e698ac9cd15fa306: Status 404 returned error can't find the container with id eace16a810da0b586127d067b7b36e79ed0c30bb897ea139e698ac9cd15fa306 Sep 13 19:58:25.507510 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:25.507490 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92652107ed38ad17b29af94e128de12.slice/crio-7bd21d9b27e94d02b6a7b86a459cf91beb2e678ff9348b533147c56097dea790 WatchSource:0}: Error finding container 7bd21d9b27e94d02b6a7b86a459cf91beb2e678ff9348b533147c56097dea790: Status 404 returned error can't find the container with id 7bd21d9b27e94d02b6a7b86a459cf91beb2e678ff9348b533147c56097dea790 Sep 13 19:58:25.511144 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.511128 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 13 19:58:25.518610 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.518595 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" Sep 13 19:58:25.529132 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.529117 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 13 19:58:25.530785 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.530771 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" Sep 13 19:58:25.541391 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.541375 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 13 19:58:25.739534 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:25.739456 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 13 19:58:26.102829 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.102747 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 13 19:58:26.295018 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.294986 2578 apiserver.go:52] "Watching apiserver" Sep 13 19:58:26.300596 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.300561 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 13 19:58:26.301080 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.301055 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal","openshift-image-registry/node-ca-xfg7g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal","openshift-multus/network-metrics-daemon-xk2rl","openshift-network-operator/iptables-alerter-j8mls","openshift-ovn-kubernetes/ovnkube-node-sq2fp","kube-system/global-pull-secret-syncer-8lrwz","kube-system/konnectivity-agent-7hslg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468","openshift-cluster-node-tuning-operator/tuned-zrmdm","openshift-dns/node-resolver-x6bb4","openshift-multus/multus-7xzwk","openshift-multus/multus-additional-cni-plugins-phbjh","openshift-network-diagnostics/network-check-target-8xd4b"] Sep 13 19:58:26.304175 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.303876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.304175 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.303965 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:26.306361 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.306309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.308457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.308430 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 13 19:58:26.308763 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.308744 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 13 19:58:26.309022 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.309004 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 13 19:58:26.309088 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.309050 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 13 19:58:26.309330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.309307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:26.309427 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.309339 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2czds\"" Sep 13 19:58:26.309427 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.309410 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 13 19:58:26.309532 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.309412 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:26.313374 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.313351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:26.313529 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.313423 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:26.317205 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.316288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.318005 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.317982 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 13 19:58:26.320108 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.320088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.320599 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.320573 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 13 19:58:26.320796 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.320776 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 13 19:58:26.320857 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.320821 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gm4vr\"" Sep 13 19:58:26.320951 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.320927 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 13 19:58:26.322360 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.322339 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 13 19:58:26.323016 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.322752 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 13 19:58:26.323016 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.322790 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 13 19:58:26.323016 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.322805 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 13 19:58:26.323016 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.322969 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 13 19:58:26.323339 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.323090 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6gvjl\"" Sep 13 19:58:26.323339 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.323199 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 13 19:58:26.323339 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.323300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.325369 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.325349 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 13 19:58:26.325739 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.325350 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 13 19:58:26.325739 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.325638 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 13 19:58:26.325739 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.325731 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lspmm\"" Sep 13 19:58:26.326256 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.326236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.328151 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.328134 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-njqq9\"" Sep 13 19:58:26.332792 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.332387 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 13 19:58:26.332792 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.332605 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 13 19:58:26.332792 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.332714 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 13 19:58:26.333016 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.332841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-var-lib-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-cni-bin\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-system-cni-dir\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktcc\" (UniqueName: \"kubernetes.io/projected/3898959b-2d2f-4923-9900-288c55d80efc-kube-api-access-fktcc\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ckp\" (UniqueName: \"kubernetes.io/projected/2473e3dd-c0cf-4159-8b9d-230c808ea682-kube-api-access-l7ckp\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-ovn\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-node-log\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-kubelet\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-host-slash\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333820 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-registration-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-device-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.334058 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-sys-fs\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7421f4bb-2546-4d47-9444-86fb057812c3-dbus\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.333979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-run-netns\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-systemd\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-log-socket\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-cni-netd\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7421f4bb-2546-4d47-9444-86fb057812c3-kubelet-config\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.334925 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.334793 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 13 19:58:26.335536 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335145 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 13 19:58:26.335536 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335214 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z84zk\"" Sep 13 19:58:26.335536 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335149 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 13 19:58:26.335536 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-slash\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.335536 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.335791 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-env-overrides\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.335791 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.335791 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-etc-selinux\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.335791 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-etc-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.335791 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.335791 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-systemd-units\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-iptables-alerter-script\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-cnibin\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335934 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhvb\" (UniqueName: \"kubernetes.io/projected/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-kube-api-access-5lhvb\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.335978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pcwv\" (UniqueName: \"kubernetes.io/projected/1af9c282-124a-4393-a66c-a54dff34e353-kube-api-access-5pcwv\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.336012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.336046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovnkube-script-lib\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.336074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovnkube-config\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.336121 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.336105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.336685 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.336136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42fk\" (UniqueName: \"kubernetes.io/projected/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-kube-api-access-s42fk\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.336685 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.336163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-socket-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.336685 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.336591 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jrfx8\"" Sep 13 19:58:26.337888 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.337298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.337888 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.337426 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 13 19:58:26.337888 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.337608 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 13 19:58:26.339633 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.339610 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 13 19:58:26.339927 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.339836 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7cb68\"" Sep 13 19:58:26.339927 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.339924 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 13 19:58:26.340356 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.340334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.342320 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.342298 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 13 19:58:26.342449 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.342430 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sftbb\"" Sep 13 19:58:26.354803 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.354710 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-79h52" Sep 13 19:58:26.362755 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.362459 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-79h52" Sep 13 19:58:26.419795 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.419743 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" event={"ID":"c92652107ed38ad17b29af94e128de12","Type":"ContainerStarted","Data":"7bd21d9b27e94d02b6a7b86a459cf91beb2e678ff9348b533147c56097dea790"} Sep 13 19:58:26.420076 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.420060 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 13 19:58:26.421682 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.421645 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" event={"ID":"419cc3454d80e28154e06a9c276d2a69","Type":"ContainerStarted","Data":"eace16a810da0b586127d067b7b36e79ed0c30bb897ea139e698ac9cd15fa306"} Sep 13 19:58:26.436900 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.436873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-cni-bin\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437006 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.436906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-var-lib-kubelet\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.437006 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.436926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-host\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.437006 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.436952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kltps\" (UniqueName: \"kubernetes.io/projected/745e61bb-9737-45ed-8405-2b2cec837d8c-kube-api-access-kltps\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.437006 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.436975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-cni-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.437006 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.436998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-cni-bin\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktcc\" (UniqueName: \"kubernetes.io/projected/3898959b-2d2f-4923-9900-288c55d80efc-kube-api-access-fktcc\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ckp\" (UniqueName: \"kubernetes.io/projected/2473e3dd-c0cf-4159-8b9d-230c808ea682-kube-api-access-l7ckp\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/745e61bb-9737-45ed-8405-2b2cec837d8c-tmp\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-host\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwt9\" (UniqueName: \"kubernetes.io/projected/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-kube-api-access-2lwt9\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-hostroot\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-registration-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-run-netns\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-kubernetes\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-systemd\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-cni-netd\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437300 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysctl-conf\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-socket-dir-parent\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-cni-multus\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-conf-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7421f4bb-2546-4d47-9444-86fb057812c3-kubelet-config\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-slash\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437620 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-cni-bin\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7421f4bb-2546-4d47-9444-86fb057812c3-kubelet-config\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437705 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-registration-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437725 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-run-netns\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-systemd\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-multus-certs\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437809 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-cni-netd\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-etc-selinux\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-etc-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.437914 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysconfig\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-systemd-units\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-cnibin\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.437997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438049 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-slash\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.438083 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438090 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-etc-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-cnibin\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438146 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-etc-selinux\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-systemd-units\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.438288 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret podName:7421f4bb-2546-4d47-9444-86fb057812c3 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:26.938174812 +0000 UTC m=+3.021319721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret") pod "global-pull-secret-syncer-8lrwz" (UID: "7421f4bb-2546-4d47-9444-86fb057812c3") : object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovnkube-script-lib\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-lib-modules\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f676cc21-4b60-4544-b5b8-1c7d297575c4-hosts-file\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-k8s-cni-cncf-io\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/343b6278-de15-4a40-9166-8a253a28bbf6-agent-certs\") pod \"konnectivity-agent-7hslg\" (UID: \"343b6278-de15-4a40-9166-8a253a28bbf6\") " pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.438580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-run\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-kubelet\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnczq\" (UniqueName: \"kubernetes.io/projected/b99c914e-69ca-4aed-9855-b34b7ef4b856-kube-api-access-vnczq\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/343b6278-de15-4a40-9166-8a253a28bbf6-konnectivity-ca\") pod \"konnectivity-agent-7hslg\" (UID: \"343b6278-de15-4a40-9166-8a253a28bbf6\") " pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-socket-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysctl-d\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-serviceca\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-cnibin\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-system-cni-dir\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-ovn\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-socket-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-node-log\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438729 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-system-cni-dir\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-ovn\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-modprobe-d\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438822 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-node-log\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f676cc21-4b60-4544-b5b8-1c7d297575c4-tmp-dir\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.439457 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-system-cni-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438918 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovnkube-script-lib\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b99c914e-69ca-4aed-9855-b34b7ef4b856-cni-binary-copy\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.438965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-kubelet\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439005 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-kubelet\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-host-slash\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-host-slash\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-device-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439235 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-run-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-tuned\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439391 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-device-dir\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:26.440325 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-sys-fs\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7421f4bb-2546-4d47-9444-86fb057812c3-dbus\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-log-socket\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-env-overrides\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2s8\" (UniqueName: \"kubernetes.io/projected/f676cc21-4b60-4544-b5b8-1c7d297575c4-kube-api-access-8j2s8\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439555 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439565 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af9c282-124a-4393-a66c-a54dff34e353-sys-fs\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3898959b-2d2f-4923-9900-288c55d80efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-log-socket\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7421f4bb-2546-4d47-9444-86fb057812c3-dbus\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.439760 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.439807 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:26.93979182 +0000 UTC m=+3.022936723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-iptables-alerter-script\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439877 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.441201 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhvb\" (UniqueName: \"kubernetes.io/projected/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-kube-api-access-5lhvb\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pcwv\" (UniqueName: \"kubernetes.io/projected/1af9c282-124a-4393-a66c-a54dff34e353-kube-api-access-5pcwv\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439941 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-env-overrides\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.439986 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-systemd\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-sys\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3898959b-2d2f-4923-9900-288c55d80efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-netns\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-daemon-config\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovnkube-config\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42fk\" (UniqueName: \"kubernetes.io/projected/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-kube-api-access-s42fk\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-etc-kubernetes\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-var-lib-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440340 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-var-lib-openvswitch\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-iptables-alerter-script\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.440716 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovnkube-config\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.442111 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.441382 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 13 19:58:26.446062 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.446041 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 13 19:58:26.446167 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.446066 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 13 19:58:26.446167 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.446092 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gkf9j for pod openshift-network-diagnostics/network-check-target-8xd4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:26.446167 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.446150 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j podName:6cd77ee5-6662-4eef-8cf1-f6f028dac132 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:26.946131717 +0000 UTC m=+3.029276631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gkf9j" (UniqueName: "kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j") pod "network-check-target-8xd4b" (UID: "6cd77ee5-6662-4eef-8cf1-f6f028dac132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:26.447906 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.447874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.449493 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.449468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktcc\" (UniqueName: \"kubernetes.io/projected/3898959b-2d2f-4923-9900-288c55d80efc-kube-api-access-fktcc\") pod \"multus-additional-cni-plugins-phbjh\" (UID: \"3898959b-2d2f-4923-9900-288c55d80efc\") " pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.449493 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.449486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhvb\" (UniqueName: \"kubernetes.io/projected/720f3bca-cf2d-41cf-81dd-c74a6c5fd581-kube-api-access-5lhvb\") pod \"iptables-alerter-j8mls\" (UID: \"720f3bca-cf2d-41cf-81dd-c74a6c5fd581\") " pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.450266 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.450017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ckp\" (UniqueName: \"kubernetes.io/projected/2473e3dd-c0cf-4159-8b9d-230c808ea682-kube-api-access-l7ckp\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:26.451116 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.451093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42fk\" (UniqueName: \"kubernetes.io/projected/1cf36c1b-4c27-46c0-acff-e80da5efc3b0-kube-api-access-s42fk\") pod \"ovnkube-node-sq2fp\" (UID: \"1cf36c1b-4c27-46c0-acff-e80da5efc3b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.451992 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.451970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pcwv\" (UniqueName: \"kubernetes.io/projected/1af9c282-124a-4393-a66c-a54dff34e353-kube-api-access-5pcwv\") pod \"aws-ebs-csi-driver-node-pz468\" (UID: \"1af9c282-124a-4393-a66c-a54dff34e353\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.541577 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/745e61bb-9737-45ed-8405-2b2cec837d8c-tmp\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.541577 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-host\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.541808 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwt9\" (UniqueName: \"kubernetes.io/projected/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-kube-api-access-2lwt9\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.541808 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-hostroot\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.541808 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-kubernetes\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.541808 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541683 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-host\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.541808 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysctl-conf\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.541808 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-socket-dir-parent\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.541808 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541782 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-hostroot\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-cni-multus\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541827 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-cni-multus\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-conf-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541817 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-kubernetes\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-socket-dir-parent\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-multus-certs\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541917 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysctl-conf\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541924 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-conf-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-multus-certs\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.541959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysconfig\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-lib-modules\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f676cc21-4b60-4544-b5b8-1c7d297575c4-hosts-file\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-k8s-cni-cncf-io\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542066 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysconfig\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/343b6278-de15-4a40-9166-8a253a28bbf6-agent-certs\") pod \"konnectivity-agent-7hslg\" (UID: \"343b6278-de15-4a40-9166-8a253a28bbf6\") " pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542139 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-k8s-cni-cncf-io\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542146 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f676cc21-4b60-4544-b5b8-1c7d297575c4-hosts-file\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.542204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-run\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542204 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-run\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-kubelet\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-lib-modules\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnczq\" (UniqueName: \"kubernetes.io/projected/b99c914e-69ca-4aed-9855-b34b7ef4b856-kube-api-access-vnczq\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/343b6278-de15-4a40-9166-8a253a28bbf6-konnectivity-ca\") pod \"konnectivity-agent-7hslg\" (UID: \"343b6278-de15-4a40-9166-8a253a28bbf6\") " pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542279 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-kubelet\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysctl-d\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-serviceca\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-cnibin\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-modprobe-d\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f676cc21-4b60-4544-b5b8-1c7d297575c4-tmp-dir\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-system-cni-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-cnibin\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b99c914e-69ca-4aed-9855-b34b7ef4b856-cni-binary-copy\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-tuned\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2s8\" (UniqueName: \"kubernetes.io/projected/f676cc21-4b60-4544-b5b8-1c7d297575c4-kube-api-access-8j2s8\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542582 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-sysctl-d\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543113 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542496 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-modprobe-d\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-systemd\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-sys\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-netns\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-daemon-config\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-etc-kubernetes\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-var-lib-kubelet\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-host\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kltps\" (UniqueName: \"kubernetes.io/projected/745e61bb-9737-45ed-8405-2b2cec837d8c-kube-api-access-kltps\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-cni-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-cni-bin\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542843 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-serviceca\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f676cc21-4b60-4544-b5b8-1c7d297575c4-tmp-dir\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542870 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/343b6278-de15-4a40-9166-8a253a28bbf6-konnectivity-ca\") pod \"konnectivity-agent-7hslg\" (UID: \"343b6278-de15-4a40-9166-8a253a28bbf6\") " pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542877 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-etc-kubernetes\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542940 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-host\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.542955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-sys\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-systemd\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.543989 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-cni-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.544875 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b99c914e-69ca-4aed-9855-b34b7ef4b856-cni-binary-copy\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.544875 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-var-lib-cni-bin\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.544875 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543118 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/745e61bb-9737-45ed-8405-2b2cec837d8c-var-lib-kubelet\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.544875 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543122 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-host-run-netns\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.544875 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b99c914e-69ca-4aed-9855-b34b7ef4b856-system-cni-dir\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.544875 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.543487 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b99c914e-69ca-4aed-9855-b34b7ef4b856-multus-daemon-config\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.546085 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.546060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/343b6278-de15-4a40-9166-8a253a28bbf6-agent-certs\") pod \"konnectivity-agent-7hslg\" (UID: \"343b6278-de15-4a40-9166-8a253a28bbf6\") " pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.546466 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.546440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/745e61bb-9737-45ed-8405-2b2cec837d8c-tmp\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.548037 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.548002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/745e61bb-9737-45ed-8405-2b2cec837d8c-etc-tuned\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.550042 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.550016 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwt9\" (UniqueName: \"kubernetes.io/projected/9746c0ef-e12e-4b7e-9b92-e3878cc381d9-kube-api-access-2lwt9\") pod \"node-ca-xfg7g\" (UID: \"9746c0ef-e12e-4b7e-9b92-e3878cc381d9\") " pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.550694 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.550669 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnczq\" (UniqueName: \"kubernetes.io/projected/b99c914e-69ca-4aed-9855-b34b7ef4b856-kube-api-access-vnczq\") pod \"multus-7xzwk\" (UID: \"b99c914e-69ca-4aed-9855-b34b7ef4b856\") " pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.551889 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.551865 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2s8\" (UniqueName: \"kubernetes.io/projected/f676cc21-4b60-4544-b5b8-1c7d297575c4-kube-api-access-8j2s8\") pod \"node-resolver-x6bb4\" (UID: \"f676cc21-4b60-4544-b5b8-1c7d297575c4\") " pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.553292 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.553268 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kltps\" (UniqueName: \"kubernetes.io/projected/745e61bb-9737-45ed-8405-2b2cec837d8c-kube-api-access-kltps\") pod \"tuned-zrmdm\" (UID: \"745e61bb-9737-45ed-8405-2b2cec837d8c\") " pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.628358 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.628282 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phbjh" Sep 13 19:58:26.641178 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.641154 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j8mls" Sep 13 19:58:26.651523 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.651502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:26.657455 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.657437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" Sep 13 19:58:26.662060 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.662041 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 13 19:58:26.664849 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.664830 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:26.673674 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.673652 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfg7g" Sep 13 19:58:26.681433 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.681411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" Sep 13 19:58:26.689167 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.689145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x6bb4" Sep 13 19:58:26.696081 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.696048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xzwk" Sep 13 19:58:26.946249 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.946156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:26.946249 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.946198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:26.946458 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:26.946283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:26.946458 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946351 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 13 19:58:26.946458 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946374 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 13 19:58:26.946458 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946374 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:26.946458 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946389 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gkf9j for pod openshift-network-diagnostics/network-check-target-8xd4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:26.946458 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946389 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:26.946458 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946443 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret podName:7421f4bb-2546-4d47-9444-86fb057812c3 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:27.946426613 +0000 UTC m=+4.029571521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret") pod "global-pull-secret-syncer-8lrwz" (UID: "7421f4bb-2546-4d47-9444-86fb057812c3") : object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:26.946731 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946464 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:27.946453494 +0000 UTC m=+4.029598389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:26.946731 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:26.946485 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j podName:6cd77ee5-6662-4eef-8cf1-f6f028dac132 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:27.946475683 +0000 UTC m=+4.029620576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkf9j" (UniqueName: "kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j") pod "network-check-target-8xd4b" (UID: "6cd77ee5-6662-4eef-8cf1-f6f028dac132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:27.366441 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:27.366354 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-12 19:53:26 +0000 UTC" deadline="2028-05-31 17:53:58.005802876 +0000 UTC" Sep 13 19:58:27.366441 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:27.366393 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15021h55m30.639414256s" Sep 13 19:58:27.413858 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:27.413821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:27.414033 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:27.413828 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:27.414033 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.413952 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:27.414180 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.414029 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:27.625797 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:27.625382 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343b6278_de15_4a40_9166_8a253a28bbf6.slice/crio-cf92cd9395b913b56681eeea93fd63bdfc219b62d42251b5e29e30fa6dceda3f WatchSource:0}: Error finding container cf92cd9395b913b56681eeea93fd63bdfc219b62d42251b5e29e30fa6dceda3f: Status 404 returned error can't find the container with id cf92cd9395b913b56681eeea93fd63bdfc219b62d42251b5e29e30fa6dceda3f Sep 13 19:58:27.628493 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:27.628383 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720f3bca_cf2d_41cf_81dd_c74a6c5fd581.slice/crio-53831d7a8afb5f42405a2061c15bc927e8223644b9696c3c80b7c4532ecd501e WatchSource:0}: Error finding container 53831d7a8afb5f42405a2061c15bc927e8223644b9696c3c80b7c4532ecd501e: Status 404 returned error can't find the container with id 53831d7a8afb5f42405a2061c15bc927e8223644b9696c3c80b7c4532ecd501e Sep 13 19:58:27.628905 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:27.628844 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf676cc21_4b60_4544_b5b8_1c7d297575c4.slice/crio-a0751e064fb68e5f31777e76d0703513c776705d662c7f5da18696043d025bb6 WatchSource:0}: Error finding container a0751e064fb68e5f31777e76d0703513c776705d662c7f5da18696043d025bb6: Status 404 returned error can't find the container with id a0751e064fb68e5f31777e76d0703513c776705d662c7f5da18696043d025bb6 Sep 13 19:58:27.629781 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:27.629750 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9746c0ef_e12e_4b7e_9b92_e3878cc381d9.slice/crio-676606c52f3ae8ae5a4792516de0de1c2298280519da084f32bec144575f33e5 WatchSource:0}: Error finding container 676606c52f3ae8ae5a4792516de0de1c2298280519da084f32bec144575f33e5: Status 404 returned error can't find the container with id 676606c52f3ae8ae5a4792516de0de1c2298280519da084f32bec144575f33e5 Sep 13 19:58:27.633475 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:27.633260 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99c914e_69ca_4aed_9855_b34b7ef4b856.slice/crio-60c98a32f96b76fe05fa1fbceb7f5d42c5bb4e000ed67b03a32669cd78bc2981 WatchSource:0}: Error finding container 60c98a32f96b76fe05fa1fbceb7f5d42c5bb4e000ed67b03a32669cd78bc2981: Status 404 returned error can't find the container with id 60c98a32f96b76fe05fa1fbceb7f5d42c5bb4e000ed67b03a32669cd78bc2981 Sep 13 19:58:27.634584 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:27.634174 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745e61bb_9737_45ed_8405_2b2cec837d8c.slice/crio-ec433889cb71e979b52610276a78ffdc131e622973df15ae297c5861a567c2f3 WatchSource:0}: Error finding container ec433889cb71e979b52610276a78ffdc131e622973df15ae297c5861a567c2f3: Status 404 returned error can't find the container with id ec433889cb71e979b52610276a78ffdc131e622973df15ae297c5861a567c2f3 Sep 13 19:58:27.636077 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:27.635939 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af9c282_124a_4393_a66c_a54dff34e353.slice/crio-5977177aa9969370d96009bdc60226c98f99c49ac25a5d8dfe8a62e3e35db9a5 WatchSource:0}: Error finding container 5977177aa9969370d96009bdc60226c98f99c49ac25a5d8dfe8a62e3e35db9a5: Status 404 returned error can't find the container with id 5977177aa9969370d96009bdc60226c98f99c49ac25a5d8dfe8a62e3e35db9a5 Sep 13 19:58:27.955695 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:27.955659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:27.955801 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:27.955722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:27.955801 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:27.955765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:27.955868 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955816 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:27.955903 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955868 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 13 19:58:27.955903 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955879 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:27.955969 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955884 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:29.955863664 +0000 UTC m=+6.039008571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:27.955969 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955884 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 13 19:58:27.955969 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955947 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret podName:7421f4bb-2546-4d47-9444-86fb057812c3 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:29.955929159 +0000 UTC m=+6.039074058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret") pod "global-pull-secret-syncer-8lrwz" (UID: "7421f4bb-2546-4d47-9444-86fb057812c3") : object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:27.955969 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955953 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gkf9j for pod openshift-network-diagnostics/network-check-target-8xd4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:27.956099 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:27.955994 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j podName:6cd77ee5-6662-4eef-8cf1-f6f028dac132 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:29.955986127 +0000 UTC m=+6.039131026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkf9j" (UniqueName: "kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j") pod "network-check-target-8xd4b" (UID: "6cd77ee5-6662-4eef-8cf1-f6f028dac132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:28.367897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.367355 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-12 19:53:26 +0000 UTC" deadline="2028-05-27 16:03:56.961606828 +0000 UTC" Sep 13 19:58:28.367897 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.367419 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14924h5m28.594192803s" Sep 13 19:58:28.417074 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.416227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:28.417074 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:28.416405 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:28.444392 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.443539 2578 generic.go:356] "Generic (PLEG): container finished" podID="419cc3454d80e28154e06a9c276d2a69" containerID="874cc00c040d7189bbd19fc947906cc5ba8fa664fb9a9a58f30b92417b8ca327" exitCode=0 Sep 13 19:58:28.444392 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.443631 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" event={"ID":"419cc3454d80e28154e06a9c276d2a69","Type":"ContainerDied","Data":"874cc00c040d7189bbd19fc947906cc5ba8fa664fb9a9a58f30b92417b8ca327"} Sep 13 19:58:28.473667 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.473556 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" event={"ID":"1af9c282-124a-4393-a66c-a54dff34e353","Type":"ContainerStarted","Data":"5977177aa9969370d96009bdc60226c98f99c49ac25a5d8dfe8a62e3e35db9a5"} Sep 13 19:58:28.480978 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.480890 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" event={"ID":"745e61bb-9737-45ed-8405-2b2cec837d8c","Type":"ContainerStarted","Data":"ec433889cb71e979b52610276a78ffdc131e622973df15ae297c5861a567c2f3"} Sep 13 19:58:28.495617 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.495587 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xzwk" event={"ID":"b99c914e-69ca-4aed-9855-b34b7ef4b856","Type":"ContainerStarted","Data":"60c98a32f96b76fe05fa1fbceb7f5d42c5bb4e000ed67b03a32669cd78bc2981"} Sep 13 19:58:28.505416 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.505385 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfg7g" event={"ID":"9746c0ef-e12e-4b7e-9b92-e3878cc381d9","Type":"ContainerStarted","Data":"676606c52f3ae8ae5a4792516de0de1c2298280519da084f32bec144575f33e5"} Sep 13 19:58:28.520283 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.519963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7hslg" event={"ID":"343b6278-de15-4a40-9166-8a253a28bbf6","Type":"ContainerStarted","Data":"cf92cd9395b913b56681eeea93fd63bdfc219b62d42251b5e29e30fa6dceda3f"} Sep 13 19:58:28.526641 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.526603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"fb456e1cb47e66d186689b0237a9632afc88ec5b55528ed1c2fecb5bdf1e1ddb"} Sep 13 19:58:28.558200 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.558148 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x6bb4" event={"ID":"f676cc21-4b60-4544-b5b8-1c7d297575c4","Type":"ContainerStarted","Data":"a0751e064fb68e5f31777e76d0703513c776705d662c7f5da18696043d025bb6"} Sep 13 19:58:28.585692 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.585658 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j8mls" event={"ID":"720f3bca-cf2d-41cf-81dd-c74a6c5fd581","Type":"ContainerStarted","Data":"53831d7a8afb5f42405a2061c15bc927e8223644b9696c3c80b7c4532ecd501e"} Sep 13 19:58:28.589699 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.589671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerStarted","Data":"c9583bdd89bf9939f1333bce79ec0ce2af0170e12e99eaa642e9419f74cf6dcf"} Sep 13 19:58:28.616193 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.616163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" event={"ID":"c92652107ed38ad17b29af94e128de12","Type":"ContainerStarted","Data":"65516a8aca563ca63668dd2052319f19351397f0d08050b1743bf17a9714bfbd"} Sep 13 19:58:28.633828 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:28.633731 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-146.ec2.internal" podStartSLOduration=2.633710549 podStartE2EDuration="2.633710549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:28.633117731 +0000 UTC m=+4.716262649" watchObservedRunningTime="2026-09-13 19:58:28.633710549 +0000 UTC m=+4.716855469" Sep 13 19:58:29.414287 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:29.413922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:29.414287 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.414093 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:29.414995 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:29.414523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:29.414995 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.414623 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:29.647266 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:29.646816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" event={"ID":"419cc3454d80e28154e06a9c276d2a69","Type":"ContainerStarted","Data":"08d9bb6d7a22707c6fe5a38970e38761e963fea66d3d4468701bd64374065fe3"} Sep 13 19:58:29.664861 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:29.664034 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-146.ec2.internal" podStartSLOduration=3.664010777 podStartE2EDuration="3.664010777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:29.663092232 +0000 UTC m=+5.746237150" watchObservedRunningTime="2026-09-13 19:58:29.664010777 +0000 UTC m=+5.747155696" Sep 13 19:58:29.973176 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:29.973081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:29.973176 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:29.973144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:29.973520 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:29.973206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:29.973520 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.973355 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:29.973520 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.973420 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret podName:7421f4bb-2546-4d47-9444-86fb057812c3 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:33.973401773 +0000 UTC m=+10.056546672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret") pod "global-pull-secret-syncer-8lrwz" (UID: "7421f4bb-2546-4d47-9444-86fb057812c3") : object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:29.973982 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.973963 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 13 19:58:29.974060 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.973993 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 13 19:58:29.974060 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.974009 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gkf9j for pod openshift-network-diagnostics/network-check-target-8xd4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:29.974060 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.974057 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j podName:6cd77ee5-6662-4eef-8cf1-f6f028dac132 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:33.974041715 +0000 UTC m=+10.057186611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkf9j" (UniqueName: "kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j") pod "network-check-target-8xd4b" (UID: "6cd77ee5-6662-4eef-8cf1-f6f028dac132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:29.974219 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.974132 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:29.974219 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:29.974170 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:33.974155789 +0000 UTC m=+10.057300698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:30.415924 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:30.414092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:30.416404 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:30.416074 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:31.414528 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:31.414493 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:31.414716 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:31.414634 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:31.414995 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:31.414975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:31.415084 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:31.415066 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:32.415089 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:32.414567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:32.415089 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:32.414721 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:33.414775 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:33.414746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:33.414934 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:33.414855 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:33.415355 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:33.415214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:33.415355 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:33.415315 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:34.005029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:34.005076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:34.005110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005240 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005299 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret podName:7421f4bb-2546-4d47-9444-86fb057812c3 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:42.005285702 +0000 UTC m=+18.088430594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret") pod "global-pull-secret-syncer-8lrwz" (UID: "7421f4bb-2546-4d47-9444-86fb057812c3") : object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005472 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005502 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005525 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005538 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:42.005520713 +0000 UTC m=+18.088665619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005538 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gkf9j for pod openshift-network-diagnostics/network-check-target-8xd4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:34.005620 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.005592 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j podName:6cd77ee5-6662-4eef-8cf1-f6f028dac132 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:42.005580396 +0000 UTC m=+18.088725289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkf9j" (UniqueName: "kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j") pod "network-check-target-8xd4b" (UID: "6cd77ee5-6662-4eef-8cf1-f6f028dac132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:34.413995 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:34.413918 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:34.414139 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:34.414070 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:35.414018 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:35.413972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:35.414496 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:35.414037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:35.414496 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:35.414127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:35.414496 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:35.414262 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:36.418278 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:36.418179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:36.418705 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:36.418335 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:37.414623 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:37.414593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:37.414623 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:37.414632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:37.414840 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:37.414716 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:37.414840 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:37.414785 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:38.414535 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:38.414364 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:38.415011 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:38.414638 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:39.414270 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:39.414237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:39.414270 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:39.414256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:39.414491 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:39.414362 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:39.414532 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:39.414500 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:40.414746 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:40.414715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:40.415179 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:40.414866 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:41.414419 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:41.414384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:41.414615 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:41.414384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:41.414615 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:41.414500 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:41.414615 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:41.414551 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:42.070002 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:42.069964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:42.070039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:42.070076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070125 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070207 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret podName:7421f4bb-2546-4d47-9444-86fb057812c3 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:58.070172819 +0000 UTC m=+34.153317713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret") pod "global-pull-secret-syncer-8lrwz" (UID: "7421f4bb-2546-4d47-9444-86fb057812c3") : object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070213 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070218 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070233 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070245 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gkf9j for pod openshift-network-diagnostics/network-check-target-8xd4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070272 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:58.070254967 +0000 UTC m=+34.153399864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:42.070428 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.070291 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j podName:6cd77ee5-6662-4eef-8cf1-f6f028dac132 nodeName:}" failed. No retries permitted until 2026-09-13 19:58:58.07028117 +0000 UTC m=+34.153426066 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkf9j" (UniqueName: "kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j") pod "network-check-target-8xd4b" (UID: "6cd77ee5-6662-4eef-8cf1-f6f028dac132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:42.414545 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:42.414472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:42.414698 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:42.414579 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:43.414780 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:43.414741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:43.415176 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:43.414741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:43.415176 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:43.414880 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:43.415176 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:43.414950 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:44.414709 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:44.414683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:44.414852 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:44.414787 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:45.413758 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:45.413721 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:45.413942 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:45.413841 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:45.413942 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:45.413906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:45.414051 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:45.414013 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:45.681965 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:45.681928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" event={"ID":"745e61bb-9737-45ed-8405-2b2cec837d8c","Type":"ContainerStarted","Data":"b7724a008e8856f74ce1c164641363df49d02b560b22aab90e67d8a32ffb8529"} Sep 13 19:58:45.691010 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:45.690761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7hslg" event={"ID":"343b6278-de15-4a40-9166-8a253a28bbf6","Type":"ContainerStarted","Data":"4470be195c10f14f1e10fef475657664e020dfdebb6fefbc29d45b5bf27a2d93"} Sep 13 19:58:45.699695 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:45.699640 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zrmdm" podStartSLOduration=2.848623575 podStartE2EDuration="20.69961946s" totalImagesPullingTime="17.850995885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:45.699031532 +0000 UTC m=+21.782176445" watchObservedRunningTime="2026-09-13 19:58:45.69961946 +0000 UTC m=+21.782764376" Sep 13 19:58:45.726750 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:45.726704 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7hslg" podStartSLOduration=10.00680508 podStartE2EDuration="20.726690064s" totalImagesPullingTime="10.719884984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:45.711530434 +0000 UTC m=+21.794675348" watchObservedRunningTime="2026-09-13 19:58:45.726690064 +0000 UTC m=+21.809834978" Sep 13 19:58:45.745073 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:45.745025 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x6bb4" podStartSLOduration=2.89073547 podStartE2EDuration="20.745008665s" totalImagesPullingTime="17.854273195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:45.727066321 +0000 UTC m=+21.810211234" watchObservedRunningTime="2026-09-13 19:58:45.745008665 +0000 UTC m=+21.828153641" Sep 13 19:58:46.414780 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.414694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:46.414936 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:46.414838 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:46.548573 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.548538 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:46.549245 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.549225 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:46.701415 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.701345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xzwk" event={"ID":"b99c914e-69ca-4aed-9855-b34b7ef4b856","Type":"ContainerStarted","Data":"12bb1a41a2febb76e86e5253928ddfb24017eff01cd49e61af34c6b279ed57f7"} Sep 13 19:58:46.702851 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.702823 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfg7g" event={"ID":"9746c0ef-e12e-4b7e-9b92-e3878cc381d9","Type":"ContainerStarted","Data":"fcefc76164e17c4dc42990d0182d8053bbe92317e2f6bd5b7b269908ae842726"} Sep 13 19:58:46.705780 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.705761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"114659f1cacd12951fd2e24d7040bcdf4d3f199d06794420f98f394cb0fbfa42"} Sep 13 19:58:46.705956 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.705819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"4f3542c7a5fdae033cdc7ea88ef3e91d6bd1b55a9a5723bc66b4c6236cdf19ee"} Sep 13 19:58:46.705956 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.705856 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"f7e11d28e60cfb383d86d11ca2d687aba50ec1def8d9cb6a5651f892d37e8b7a"} Sep 13 19:58:46.705956 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.705901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"a19412d48df0fa641f65a7bc6be166adcbe96b314213217bd6e23770f2f16a28"} Sep 13 19:58:46.705956 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.705940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"b71c2dd126a82fc0b126853afa7bb7ac2348c607786c5a0378d692127f5eaae8"} Sep 13 19:58:46.706765 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.705983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"acb9d78c35503725eae72f4ba5afc352582328487692cd910efa13c9b40f5045"} Sep 13 19:58:46.708248 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.707963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x6bb4" event={"ID":"f676cc21-4b60-4544-b5b8-1c7d297575c4","Type":"ContainerStarted","Data":"c1df1803c14fda37eeda1f6d0b5a485ba5687a3a122deedd148da5a98ede9635"} Sep 13 19:58:46.710212 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.710178 2578 generic.go:356] "Generic (PLEG): container finished" podID="3898959b-2d2f-4923-9900-288c55d80efc" containerID="d82c5363b49e73a7ca766286bb67b9fdd96ad5fb929283715bcd768b268e6c53" exitCode=0 Sep 13 19:58:46.710291 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.710213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerDied","Data":"d82c5363b49e73a7ca766286bb67b9fdd96ad5fb929283715bcd768b268e6c53"} Sep 13 19:58:46.711636 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.711610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" event={"ID":"1af9c282-124a-4393-a66c-a54dff34e353","Type":"ContainerStarted","Data":"71a8ff8691523b3eb405e7f88adcffae407bf8129a3b77c413021fe14ec07361"} Sep 13 19:58:46.711994 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.711972 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:46.712318 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.712305 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7hslg" Sep 13 19:58:46.715275 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.715231 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7xzwk" podStartSLOduration=3.813375232 podStartE2EDuration="21.715219113s" totalImagesPullingTime="17.901843881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:46.715089599 +0000 UTC m=+22.798234512" watchObservedRunningTime="2026-09-13 19:58:46.715219113 +0000 UTC m=+22.798364039" Sep 13 19:58:46.769100 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.769048 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xfg7g" podStartSLOduration=3.915848121 podStartE2EDuration="21.769030167s" totalImagesPullingTime="17.853182046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:46.769010331 +0000 UTC m=+22.852155300" watchObservedRunningTime="2026-09-13 19:58:46.769030167 +0000 UTC m=+22.852175082" Sep 13 19:58:46.839519 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:46.839380 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 13 19:58:47.397647 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.397505 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-13T19:58:46.839516082Z","UUID":"88e91334-7f55-430c-9fac-727b6ab61847","Handler":null,"Name":"","Endpoint":""} Sep 13 19:58:47.398975 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.398955 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 13 19:58:47.399089 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.398990 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 13 19:58:47.414157 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.414133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:47.414157 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.414156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:47.414329 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:47.414254 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:47.414425 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:47.414403 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:47.716643 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.716514 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j8mls" event={"ID":"720f3bca-cf2d-41cf-81dd-c74a6c5fd581","Type":"ContainerStarted","Data":"dce4c8c644f23e1bdd3f30d38eab1c2d4d2f97f1c0b6d7980114209df157eec5"} Sep 13 19:58:47.718780 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.718754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" event={"ID":"1af9c282-124a-4393-a66c-a54dff34e353","Type":"ContainerStarted","Data":"d787a1fef297d3fa56bd01806c0f0c40efc39e5d76d73d69a6cbc97832123353"} Sep 13 19:58:47.729934 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.729880 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j8mls" podStartSLOduration=4.875121953 podStartE2EDuration="22.729866586s" totalImagesPullingTime="17.854744633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:47.729326145 +0000 UTC m=+23.812471072" watchObservedRunningTime="2026-09-13 19:58:47.729866586 +0000 UTC m=+23.813011525" Sep 13 19:58:47.871282 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.871244 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:47.951961 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:47.951931 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:48.414377 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:48.414302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:48.414532 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:48.414448 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:48.725872 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:48.725787 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"29936051c3c54322dcfda7afeae7db596c272f17d3919de116d1c38330c14706"} Sep 13 19:58:48.728161 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:48.728129 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" event={"ID":"1af9c282-124a-4393-a66c-a54dff34e353","Type":"ContainerStarted","Data":"bb333eefd6a071af56e7ac13b0a4bb5cab4139756b77215df08eabb9f5a64c68"} Sep 13 19:58:48.744031 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:48.743989 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pz468" podStartSLOduration=3.69726644 podStartE2EDuration="23.743975318s" totalImagesPullingTime="20.046708878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:48.743399553 +0000 UTC m=+24.826544467" watchObservedRunningTime="2026-09-13 19:58:48.743975318 +0000 UTC m=+24.827120231" Sep 13 19:58:49.414134 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:49.414102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:49.414337 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:49.414240 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:49.414337 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:49.414274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:49.414442 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:49.414379 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:50.413896 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:50.413863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:50.414345 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:50.413981 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:51.414738 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.414517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:51.415062 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.414518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:51.415062 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:51.414768 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:51.415062 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:51.414836 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:51.737148 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.737078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" event={"ID":"1cf36c1b-4c27-46c0-acff-e80da5efc3b0","Type":"ContainerStarted","Data":"85dec2d2f41d2bedb953f7eaf597431e13166a76cb534c25e6e7ffc3adcba740"} Sep 13 19:58:51.737373 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.737355 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:51.738970 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.738948 2578 generic.go:356] "Generic (PLEG): container finished" podID="3898959b-2d2f-4923-9900-288c55d80efc" containerID="70eb9e62e5d9d0ad4713bf6862fb5e965031be69adfa225b21a08b02c833756f" exitCode=0 Sep 13 19:58:51.739067 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.738977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerDied","Data":"70eb9e62e5d9d0ad4713bf6862fb5e965031be69adfa225b21a08b02c833756f"} Sep 13 19:58:51.752698 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.752679 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:51.761330 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.761292 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" podStartSLOduration=8.856354428 podStartE2EDuration="26.761280895s" totalImagesPullingTime="17.904926467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:58:51.760305625 +0000 UTC m=+27.843450538" watchObservedRunningTime="2026-09-13 19:58:51.761280895 +0000 UTC m=+27.844425808" Sep 13 19:58:51.764857 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.764837 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:51.815569 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.815542 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:51.867089 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:51.867063 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:52.413804 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.413774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:52.413990 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:52.413910 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:52.743489 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.743456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerStarted","Data":"6ded6b36e691a9ba54b13a34948ef23ea8c502feeca06ed0f88eebbd4bf875a8"} Sep 13 19:58:52.744091 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.744070 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:52.744153 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.744143 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:52.758877 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.758855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:58:52.779473 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.779449 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:52.834933 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.834764 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:52.912007 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.911965 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8lrwz"] Sep 13 19:58:52.912157 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.912103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:52.912257 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:52.912231 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:52.914950 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.914925 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8xd4b"] Sep 13 19:58:52.915067 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.915028 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:52.915151 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:52.915127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:52.915483 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.915464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk2rl"] Sep 13 19:58:52.915574 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:52.915563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:52.915687 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:52.915663 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:53.747512 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:53.747480 2578 generic.go:356] "Generic (PLEG): container finished" podID="3898959b-2d2f-4923-9900-288c55d80efc" containerID="6ded6b36e691a9ba54b13a34948ef23ea8c502feeca06ed0f88eebbd4bf875a8" exitCode=0 Sep 13 19:58:53.747864 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:53.747548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerDied","Data":"6ded6b36e691a9ba54b13a34948ef23ea8c502feeca06ed0f88eebbd4bf875a8"} Sep 13 19:58:54.414725 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:54.414698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:54.414866 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:54.414814 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:54.415282 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:54.415264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:54.415378 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:54.415359 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:54.415526 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:54.415511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:54.415643 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:54.415623 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:54.752290 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:54.752221 2578 generic.go:356] "Generic (PLEG): container finished" podID="3898959b-2d2f-4923-9900-288c55d80efc" containerID="ab5fa3bfeeb5a585cd0e5ebac04d5dd7107d0e018b0bc3b6e603b2b67872530d" exitCode=0 Sep 13 19:58:54.752597 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:54.752296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerDied","Data":"ab5fa3bfeeb5a585cd0e5ebac04d5dd7107d0e018b0bc3b6e603b2b67872530d"} Sep 13 19:58:56.414475 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:56.414433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:56.414964 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:56.414443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:56.414964 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:56.414558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:56.414964 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:56.414553 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:56.414964 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:56.414640 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:56.414964 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:56.414757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:57.823217 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:57.823167 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:57.883351 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:57.883318 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:58.091679 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.091597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:58.091679 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.091640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:58.091904 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.091701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:58.091904 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091768 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 13 19:58:58.091904 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091784 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 13 19:58:58.091904 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091795 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gkf9j for pod openshift-network-diagnostics/network-check-target-8xd4b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:58.091904 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091820 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:58.091904 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091838 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j podName:6cd77ee5-6662-4eef-8cf1-f6f028dac132 nodeName:}" failed. No retries permitted until 2026-09-13 19:59:30.091826855 +0000 UTC m=+66.174971747 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gkf9j" (UniqueName: "kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j") pod "network-check-target-8xd4b" (UID: "6cd77ee5-6662-4eef-8cf1-f6f028dac132") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 13 19:58:58.091904 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091872 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 19:59:30.09185908 +0000 UTC m=+66.175003972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 13 19:58:58.092296 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091922 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:58.092296 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.091957 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret podName:7421f4bb-2546-4d47-9444-86fb057812c3 nodeName:}" failed. No retries permitted until 2026-09-13 19:59:30.091950192 +0000 UTC m=+66.175095085 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret") pod "global-pull-secret-syncer-8lrwz" (UID: "7421f4bb-2546-4d47-9444-86fb057812c3") : object "kube-system"/"original-pull-secret" not registered Sep 13 19:58:58.414015 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.413970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:58:58.414204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.414026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:58:58.414204 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.413973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:58:58.414204 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.414099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8xd4b" podUID="6cd77ee5-6662-4eef-8cf1-f6f028dac132" Sep 13 19:58:58.414374 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.414206 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 19:58:58.414374 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:58.414295 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lrwz" podUID="7421f4bb-2546-4d47-9444-86fb057812c3" Sep 13 19:58:58.815285 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.815216 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:58.862868 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:58.862844 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:58:59.213676 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.213649 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-146.ec2.internal" event="NodeReady" Sep 13 19:58:59.213837 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.213766 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 13 19:58:59.251820 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.251792 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jcz7f"] Sep 13 19:58:59.266580 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.266553 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-89mtv"] Sep 13 19:58:59.266717 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.266699 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.268962 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.268941 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 13 19:58:59.269090 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.268974 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 13 19:58:59.269090 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.268981 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pm8n4\"" Sep 13 19:58:59.280493 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.280475 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jcz7f"] Sep 13 19:58:59.280599 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.280530 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-89mtv"] Sep 13 19:58:59.280641 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.280590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.282650 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.282630 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 13 19:58:59.282810 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.282674 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 13 19:58:59.282810 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.282753 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-stlr4\"" Sep 13 19:58:59.282956 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.282942 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 13 19:58:59.400648 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.400556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd881c97-f6a3-46b3-922c-451a18fdfafd-tmp-dir\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.400648 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.400590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjcf\" (UniqueName: \"kubernetes.io/projected/fd881c97-f6a3-46b3-922c-451a18fdfafd-kube-api-access-grjcf\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.400648 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.400614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88db34e6-cfe4-4767-9054-ba703ed75960-cert\") pod \"ingress-canary-89mtv\" (UID: \"88db34e6-cfe4-4767-9054-ba703ed75960\") " pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.400648 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.400649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd881c97-f6a3-46b3-922c-451a18fdfafd-config-volume\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.400930 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.400720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkxq\" (UniqueName: \"kubernetes.io/projected/88db34e6-cfe4-4767-9054-ba703ed75960-kube-api-access-rpkxq\") pod \"ingress-canary-89mtv\" (UID: \"88db34e6-cfe4-4767-9054-ba703ed75960\") " pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.400930 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.400774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.502023 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.501987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd881c97-f6a3-46b3-922c-451a18fdfafd-config-volume\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.502160 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.502040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkxq\" (UniqueName: \"kubernetes.io/projected/88db34e6-cfe4-4767-9054-ba703ed75960-kube-api-access-rpkxq\") pod \"ingress-canary-89mtv\" (UID: \"88db34e6-cfe4-4767-9054-ba703ed75960\") " pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.502212 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.502178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.502252 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.502232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd881c97-f6a3-46b3-922c-451a18fdfafd-tmp-dir\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.502285 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.502253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjcf\" (UniqueName: \"kubernetes.io/projected/fd881c97-f6a3-46b3-922c-451a18fdfafd-kube-api-access-grjcf\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.502314 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.502291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88db34e6-cfe4-4767-9054-ba703ed75960-cert\") pod \"ingress-canary-89mtv\" (UID: \"88db34e6-cfe4-4767-9054-ba703ed75960\") " pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.502349 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:59.502327 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 19:58:59.502392 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:58:59.502386 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 19:59:00.002371195 +0000 UTC m=+36.085516091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 19:58:59.502730 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.502708 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd881c97-f6a3-46b3-922c-451a18fdfafd-tmp-dir\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.502806 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.502789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd881c97-f6a3-46b3-922c-451a18fdfafd-config-volume\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.506304 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.506125 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88db34e6-cfe4-4767-9054-ba703ed75960-cert\") pod \"ingress-canary-89mtv\" (UID: \"88db34e6-cfe4-4767-9054-ba703ed75960\") " pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.510074 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.510054 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjcf\" (UniqueName: \"kubernetes.io/projected/fd881c97-f6a3-46b3-922c-451a18fdfafd-kube-api-access-grjcf\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:58:59.510168 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.510140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkxq\" (UniqueName: \"kubernetes.io/projected/88db34e6-cfe4-4767-9054-ba703ed75960-kube-api-access-rpkxq\") pod \"ingress-canary-89mtv\" (UID: \"88db34e6-cfe4-4767-9054-ba703ed75960\") " pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.589128 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.589100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-89mtv" Sep 13 19:58:59.767784 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.767743 2578 generic.go:356] "Generic (PLEG): container finished" podID="3898959b-2d2f-4923-9900-288c55d80efc" containerID="41a5e835aaf46f3fdabbe8a394420275351dfd6c9e51dbc925c57df9cb624c8d" exitCode=0 Sep 13 19:58:59.767784 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.767782 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerDied","Data":"41a5e835aaf46f3fdabbe8a394420275351dfd6c9e51dbc925c57df9cb624c8d"} Sep 13 19:58:59.909909 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:58:59.909879 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-89mtv"] Sep 13 19:58:59.913132 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:58:59.913106 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88db34e6_cfe4_4767_9054_ba703ed75960.slice/crio-68ee7bab4883ab3c9afa4039371570143aec8199bb3530bb4fe0b59d81ea0f25 WatchSource:0}: Error finding container 68ee7bab4883ab3c9afa4039371570143aec8199bb3530bb4fe0b59d81ea0f25: Status 404 returned error can't find the container with id 68ee7bab4883ab3c9afa4039371570143aec8199bb3530bb4fe0b59d81ea0f25 Sep 13 19:59:00.007229 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.007199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:59:00.007333 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:00.007295 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 19:59:00.007369 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:00.007357 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 19:59:01.007340772 +0000 UTC m=+37.090485676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 19:59:00.414511 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.414425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:59:00.414671 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.414425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:59:00.414671 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.414432 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:59:00.417162 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.417141 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvn9g\"" Sep 13 19:59:00.417162 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.417160 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 13 19:59:00.417411 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.417394 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bxmhk\"" Sep 13 19:59:00.417486 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.417432 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 13 19:59:00.417486 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.417466 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 13 19:59:00.417642 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.417538 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 13 19:59:00.774258 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.774169 2578 generic.go:356] "Generic (PLEG): container finished" podID="3898959b-2d2f-4923-9900-288c55d80efc" containerID="fe5a7b9b2414b2204d8a4e31d2d6bb5a4abb4ef84d1e1109e8284e6e2a7b75e0" exitCode=0 Sep 13 19:59:00.774258 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.774214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerDied","Data":"fe5a7b9b2414b2204d8a4e31d2d6bb5a4abb4ef84d1e1109e8284e6e2a7b75e0"} Sep 13 19:59:00.775787 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:00.775759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-89mtv" event={"ID":"88db34e6-cfe4-4767-9054-ba703ed75960","Type":"ContainerStarted","Data":"68ee7bab4883ab3c9afa4039371570143aec8199bb3530bb4fe0b59d81ea0f25"} Sep 13 19:59:01.017102 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:01.017065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:59:01.017670 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:01.017240 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 19:59:01.017670 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:01.017325 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 19:59:03.017305585 +0000 UTC m=+39.100450479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 19:59:01.782142 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:01.781949 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phbjh" event={"ID":"3898959b-2d2f-4923-9900-288c55d80efc","Type":"ContainerStarted","Data":"1cac61c74232b8ef1e50738ae84d614436c230d87af26748005939ddc38d9774"} Sep 13 19:59:01.802400 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:01.802129 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-phbjh" podStartSLOduration=9.072011383 podStartE2EDuration="36.802108882s" totalImagesPullingTime="27.730097499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:59:01.800775858 +0000 UTC m=+37.883920785" watchObservedRunningTime="2026-09-13 19:59:01.802108882 +0000 UTC m=+37.885253812" Sep 13 19:59:01.919765 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:01.919735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:02.019890 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:02.019854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:02.785463 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:02.785432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-89mtv" event={"ID":"88db34e6-cfe4-4767-9054-ba703ed75960","Type":"ContainerStarted","Data":"0fae979182797982d2a7aa9fcc9dbda402ed21275265d8cc2edd7694bdd29a73"} Sep 13 19:59:02.800046 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:02.799993 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-89mtv" podStartSLOduration=1.412727001 podStartE2EDuration="3.799975959s" totalImagesPullingTime="2.387248958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 19:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:59:02.798765522 +0000 UTC m=+38.881910435" watchObservedRunningTime="2026-09-13 19:59:02.799975959 +0000 UTC m=+38.883120873" Sep 13 19:59:03.033828 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:03.033785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:59:03.034247 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:03.033934 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 19:59:03.034247 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:03.034005 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 19:59:07.033982471 +0000 UTC m=+43.117127363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 19:59:03.941902 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:03.941860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:03.990728 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:03.990702 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:07.059961 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:07.059921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:59:07.060376 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:07.060054 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 19:59:07.060376 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:07.060107 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 19:59:15.060094151 +0000 UTC m=+51.143239042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 19:59:12.771481 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:12.771449 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:12.830385 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:12.830355 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:15.109904 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:15.109864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:59:15.110366 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:15.110003 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 19:59:15.110366 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:15.110063 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 19:59:31.110047489 +0000 UTC m=+67.193192381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 19:59:19.786020 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:19.785974 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:19.836854 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:19.836830 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:24.763059 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:24.763030 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sq2fp" Sep 13 19:59:24.876004 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:24.875965 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:24.928199 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:24.928157 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:27.377178 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:27.377137 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:27.431146 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:27.431110 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:30.110380 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.110336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 19:59:30.110840 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.110396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:59:30.110840 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.110426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:59:30.113027 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.113002 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 13 19:59:30.113128 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.113076 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 13 19:59:30.113128 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.113088 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 13 19:59:30.120769 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:30.120753 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 13 19:59:30.120859 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:30.120833 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 20:00:34.120811663 +0000 UTC m=+130.203956556 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : secret "metrics-daemon-secret" not found Sep 13 19:59:30.122348 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.122333 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 13 19:59:30.123683 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.123668 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7421f4bb-2546-4d47-9444-86fb057812c3-original-pull-secret\") pod \"global-pull-secret-syncer-8lrwz\" (UID: \"7421f4bb-2546-4d47-9444-86fb057812c3\") " pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:59:30.133046 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.133025 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkf9j\" (UniqueName: \"kubernetes.io/projected/6cd77ee5-6662-4eef-8cf1-f6f028dac132-kube-api-access-gkf9j\") pod \"network-check-target-8xd4b\" (UID: \"6cd77ee5-6662-4eef-8cf1-f6f028dac132\") " pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:59:30.136511 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.136494 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bxmhk\"" Sep 13 19:59:30.141092 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.141078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lrwz" Sep 13 19:59:30.145229 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.145214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:59:30.433806 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.433736 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8lrwz"] Sep 13 19:59:30.436681 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:59:30.436652 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7421f4bb_2546_4d47_9444_86fb057812c3.slice/crio-9961e30d7c8bfea7ab0014c528d77fb537e9bdf87ef7853415098c62ad56594e WatchSource:0}: Error finding container 9961e30d7c8bfea7ab0014c528d77fb537e9bdf87ef7853415098c62ad56594e: Status 404 returned error can't find the container with id 9961e30d7c8bfea7ab0014c528d77fb537e9bdf87ef7853415098c62ad56594e Sep 13 19:59:30.453718 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.453694 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8xd4b"] Sep 13 19:59:30.457247 ip-10-0-141-146 kubenswrapper[2578]: W0913 19:59:30.457219 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd77ee5_6662_4eef_8cf1_f6f028dac132.slice/crio-451208952c342442b5d1ad2af8f18e108e4385cef23ed53114576f0603b5b36a WatchSource:0}: Error finding container 451208952c342442b5d1ad2af8f18e108e4385cef23ed53114576f0603b5b36a: Status 404 returned error can't find the container with id 451208952c342442b5d1ad2af8f18e108e4385cef23ed53114576f0603b5b36a Sep 13 19:59:30.851499 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.851453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8lrwz" event={"ID":"7421f4bb-2546-4d47-9444-86fb057812c3","Type":"ContainerStarted","Data":"9961e30d7c8bfea7ab0014c528d77fb537e9bdf87ef7853415098c62ad56594e"} Sep 13 19:59:30.853140 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:30.853112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8xd4b" event={"ID":"6cd77ee5-6662-4eef-8cf1-f6f028dac132","Type":"ContainerStarted","Data":"451208952c342442b5d1ad2af8f18e108e4385cef23ed53114576f0603b5b36a"} Sep 13 19:59:31.118497 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:31.118407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 19:59:31.118942 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:31.118581 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 19:59:31.118942 ip-10-0-141-146 kubenswrapper[2578]: E0913 19:59:31.118676 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 20:00:03.118652461 +0000 UTC m=+99.201797357 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 19:59:34.866520 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:34.866481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8xd4b" event={"ID":"6cd77ee5-6662-4eef-8cf1-f6f028dac132","Type":"ContainerStarted","Data":"e8257f8486a2f6c820116267a66c76f93f255bc2e768dbff17806133b1d24b0e"} Sep 13 19:59:34.866973 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:34.866655 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 19:59:34.868027 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:34.868003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8lrwz" event={"ID":"7421f4bb-2546-4d47-9444-86fb057812c3","Type":"ContainerStarted","Data":"2c12b68999e42239b2ac119661ddb340d42b52c57288e72af6cb5fa953eadb63"} Sep 13 19:59:34.880242 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:34.880203 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8xd4b" podStartSLOduration=65.609230792 podStartE2EDuration="1m9.880175351s" totalImagesPullingTime="4.270944559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:59:34.879811247 +0000 UTC m=+70.962956161" watchObservedRunningTime="2026-09-13 19:59:34.880175351 +0000 UTC m=+70.963320266" Sep 13 19:59:34.893137 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:34.893086 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8lrwz" podStartSLOduration=65.597974369 podStartE2EDuration="1m9.893068991s" totalImagesPullingTime="4.295094622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 19:59:34.891943671 +0000 UTC m=+70.975088586" watchObservedRunningTime="2026-09-13 19:59:34.893068991 +0000 UTC m=+70.976213905" Sep 13 19:59:57.547433 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:57.547309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 19:59:57.595118 ip-10-0-141-146 kubenswrapper[2578]: I0913 19:59:57.595085 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:00:03.134011 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:03.133958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 20:00:03.134536 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:00:03.134090 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 20:00:03.134536 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:00:03.134143 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 20:01:07.134130223 +0000 UTC m=+163.217275116 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 20:00:05.872973 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:05.872942 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8xd4b" Sep 13 20:00:34.138575 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:34.138518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 20:00:34.139160 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:00:34.138688 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 13 20:00:34.139160 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:00:34.138784 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:36.138763295 +0000 UTC m=+252.221908206 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : secret "metrics-daemon-secret" not found Sep 13 20:00:42.241112 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.241074 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr"] Sep 13 20:00:42.244080 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.244054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.245009 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.244981 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd"] Sep 13 20:00:42.247496 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.247480 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.247939 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.247920 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 13 20:00:42.248871 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.248852 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 13 20:00:42.248961 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.248904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 13 20:00:42.248961 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.248915 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 13 20:00:42.251022 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.250999 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 13 20:00:42.255885 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.255865 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g8cxj\"" Sep 13 20:00:42.275730 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.275702 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr"] Sep 13 20:00:42.300444 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.300418 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd"] Sep 13 20:00:42.394789 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.394749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2793184-45a8-47a3-9fc4-d740a487a5b5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-95fc88884-knnkd\" (UID: \"e2793184-45a8-47a3-9fc4-d740a487a5b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.394789 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.394789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/769ea3eb-d85d-4ccc-85c2-9482ac32f837-tmp\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.394986 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.394824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8h7g\" (UniqueName: \"kubernetes.io/projected/769ea3eb-d85d-4ccc-85c2-9482ac32f837-kube-api-access-h8h7g\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.394986 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.394841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvxg\" (UniqueName: \"kubernetes.io/projected/e2793184-45a8-47a3-9fc4-d740a487a5b5-kube-api-access-mmvxg\") pod \"managed-serviceaccount-addon-agent-95fc88884-knnkd\" (UID: \"e2793184-45a8-47a3-9fc4-d740a487a5b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.394986 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.394879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/769ea3eb-d85d-4ccc-85c2-9482ac32f837-klusterlet-config\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.495738 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.495651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8h7g\" (UniqueName: \"kubernetes.io/projected/769ea3eb-d85d-4ccc-85c2-9482ac32f837-kube-api-access-h8h7g\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.495738 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.495688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvxg\" (UniqueName: \"kubernetes.io/projected/e2793184-45a8-47a3-9fc4-d740a487a5b5-kube-api-access-mmvxg\") pod \"managed-serviceaccount-addon-agent-95fc88884-knnkd\" (UID: \"e2793184-45a8-47a3-9fc4-d740a487a5b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.495738 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.495729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/769ea3eb-d85d-4ccc-85c2-9482ac32f837-klusterlet-config\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.495959 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.495754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2793184-45a8-47a3-9fc4-d740a487a5b5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-95fc88884-knnkd\" (UID: \"e2793184-45a8-47a3-9fc4-d740a487a5b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.495959 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.495773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/769ea3eb-d85d-4ccc-85c2-9482ac32f837-tmp\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.496266 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.496242 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/769ea3eb-d85d-4ccc-85c2-9482ac32f837-tmp\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.498494 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.498471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/769ea3eb-d85d-4ccc-85c2-9482ac32f837-klusterlet-config\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.498590 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.498471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2793184-45a8-47a3-9fc4-d740a487a5b5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-95fc88884-knnkd\" (UID: \"e2793184-45a8-47a3-9fc4-d740a487a5b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.503120 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.503094 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8h7g\" (UniqueName: \"kubernetes.io/projected/769ea3eb-d85d-4ccc-85c2-9482ac32f837-kube-api-access-h8h7g\") pod \"klusterlet-addon-workmgr-9f5885746-s7vlr\" (UID: \"769ea3eb-d85d-4ccc-85c2-9482ac32f837\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.503248 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.503229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvxg\" (UniqueName: \"kubernetes.io/projected/e2793184-45a8-47a3-9fc4-d740a487a5b5-kube-api-access-mmvxg\") pod \"managed-serviceaccount-addon-agent-95fc88884-knnkd\" (UID: \"e2793184-45a8-47a3-9fc4-d740a487a5b5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.554154 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.554120 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:42.568214 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.568177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" Sep 13 20:00:42.821319 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.821287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr"] Sep 13 20:00:42.823861 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:00:42.823827 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769ea3eb_d85d_4ccc_85c2_9482ac32f837.slice/crio-b4149cfdd26194723c1f96a3aa64a12fc072c08b50c256578f34da82ac862fef WatchSource:0}: Error finding container b4149cfdd26194723c1f96a3aa64a12fc072c08b50c256578f34da82ac862fef: Status 404 returned error can't find the container with id b4149cfdd26194723c1f96a3aa64a12fc072c08b50c256578f34da82ac862fef Sep 13 20:00:42.841091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:42.841065 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd"] Sep 13 20:00:42.843690 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:00:42.843664 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2793184_45a8_47a3_9fc4_d740a487a5b5.slice/crio-6df1bc4b40d0378f94e0ae4f72f7108b74c3f838ee7f3b3e2fc20cc73fc0b9fb WatchSource:0}: Error finding container 6df1bc4b40d0378f94e0ae4f72f7108b74c3f838ee7f3b3e2fc20cc73fc0b9fb: Status 404 returned error can't find the container with id 6df1bc4b40d0378f94e0ae4f72f7108b74c3f838ee7f3b3e2fc20cc73fc0b9fb Sep 13 20:00:43.033414 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:43.032782 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" event={"ID":"769ea3eb-d85d-4ccc-85c2-9482ac32f837","Type":"ContainerStarted","Data":"b4149cfdd26194723c1f96a3aa64a12fc072c08b50c256578f34da82ac862fef"} Sep 13 20:00:43.034550 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:43.034521 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" event={"ID":"e2793184-45a8-47a3-9fc4-d740a487a5b5","Type":"ContainerStarted","Data":"6df1bc4b40d0378f94e0ae4f72f7108b74c3f838ee7f3b3e2fc20cc73fc0b9fb"} Sep 13 20:00:48.050626 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:48.050589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" event={"ID":"769ea3eb-d85d-4ccc-85c2-9482ac32f837","Type":"ContainerStarted","Data":"4c444a74f13714ef75e82aedbe8de53f1d5814a273a3a331508ea4b71e798c4c"} Sep 13 20:00:48.051070 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:48.050767 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:48.052121 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:48.052088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" event={"ID":"e2793184-45a8-47a3-9fc4-d740a487a5b5","Type":"ContainerStarted","Data":"6844859d6559f5223bb88e745e2e869359468fea8307ad94f3ec6755efa590f4"} Sep 13 20:00:48.052578 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:48.052559 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:00:48.065802 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:48.065756 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" podStartSLOduration=0.99277289 podStartE2EDuration="6.065742513s" totalImagesPullingTime="5.072969623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:00:48.064494751 +0000 UTC m=+144.147639665" watchObservedRunningTime="2026-09-13 20:00:48.065742513 +0000 UTC m=+144.148887426" Sep 13 20:00:48.080374 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:00:48.078007 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" podStartSLOduration=1.015977984 podStartE2EDuration="6.077988933s" totalImagesPullingTime="5.062010949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:00:48.076469653 +0000 UTC m=+144.159614567" watchObservedRunningTime="2026-09-13 20:00:48.077988933 +0000 UTC m=+144.161133850" Sep 13 20:01:02.276059 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:01:02.276013 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jcz7f" podUID="fd881c97-f6a3-46b3-922c-451a18fdfafd" Sep 13 20:01:03.088413 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:01:03.088379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jcz7f" Sep 13 20:01:03.426946 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:01:03.426863 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 20:01:07.176689 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:01:07.176645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 20:01:07.177171 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:01:07.176783 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 13 20:01:07.177171 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:01:07.176849 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls podName:fd881c97-f6a3-46b3-922c-451a18fdfafd nodeName:}" failed. No retries permitted until 2026-09-13 20:03:09.176831934 +0000 UTC m=+285.259976830 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls") pod "dns-default-jcz7f" (UID: "fd881c97-f6a3-46b3-922c-451a18fdfafd") : secret "dns-default-metrics-tls" not found Sep 13 20:01:15.414162 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:01:15.414120 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 20:02:05.993629 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:05.993595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:36.168486 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.168437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 20:02:36.168963 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:36.168561 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 13 20:02:36.168963 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:36.168618 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs podName:2473e3dd-c0cf-4159-8b9d-230c808ea682 nodeName:}" failed. No retries permitted until 2026-09-13 20:04:38.168603711 +0000 UTC m=+374.251748603 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs") pod "network-metrics-daemon-xk2rl" (UID: "2473e3dd-c0cf-4159-8b9d-230c808ea682") : secret "metrics-daemon-secret" not found Sep 13 20:02:36.755862 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.755829 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh"] Sep 13 20:02:36.758850 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.758815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" Sep 13 20:02:36.759332 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.759305 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs"] Sep 13 20:02:36.761318 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.761298 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jk6sm\"" Sep 13 20:02:36.761408 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.761299 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 13 20:02:36.761466 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.761301 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 13 20:02:36.761988 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.761971 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-shgsc"] Sep 13 20:02:36.762102 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.762087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" Sep 13 20:02:36.764148 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.764131 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rctks\"" Sep 13 20:02:36.764629 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.764607 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.765934 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.765916 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh"] Sep 13 20:02:36.766613 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.766581 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 13 20:02:36.766613 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.766608 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ncp6d\"" Sep 13 20:02:36.766806 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.766714 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 13 20:02:36.766910 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.766868 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 13 20:02:36.766971 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.766936 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 13 20:02:36.771495 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.771428 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 13 20:02:36.773256 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.773227 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs"] Sep 13 20:02:36.774258 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.774240 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-shgsc"] Sep 13 20:02:36.872623 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f476a24-96a1-4723-ba1d-45bf3df32556-snapshots\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.872623 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f476a24-96a1-4723-ba1d-45bf3df32556-trusted-ca-bundle\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.872839 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f476a24-96a1-4723-ba1d-45bf3df32556-service-ca-bundle\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.872839 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f476a24-96a1-4723-ba1d-45bf3df32556-tmp\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.872839 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f476a24-96a1-4723-ba1d-45bf3df32556-serving-cert\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.872935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6pr2\" (UniqueName: \"kubernetes.io/projected/f9228fd1-3e06-48cc-9d0c-8168c74ae4ea-kube-api-access-n6pr2\") pod \"volume-data-source-validator-86695b7cc7-ps4lh\" (UID: \"f9228fd1-3e06-48cc-9d0c-8168c74ae4ea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" Sep 13 20:02:36.872935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9f5m\" (UniqueName: \"kubernetes.io/projected/1f476a24-96a1-4723-ba1d-45bf3df32556-kube-api-access-t9f5m\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.872935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.872879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2bx\" (UniqueName: \"kubernetes.io/projected/0940fe8c-22f7-42c0-8323-86799a10fb20-kube-api-access-vq2bx\") pod \"network-check-source-6f8b768455-5bnrs\" (UID: \"0940fe8c-22f7-42c0-8323-86799a10fb20\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" Sep 13 20:02:36.886489 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.886455 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:36.973620 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f476a24-96a1-4723-ba1d-45bf3df32556-serving-cert\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.973731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6pr2\" (UniqueName: \"kubernetes.io/projected/f9228fd1-3e06-48cc-9d0c-8168c74ae4ea-kube-api-access-n6pr2\") pod \"volume-data-source-validator-86695b7cc7-ps4lh\" (UID: \"f9228fd1-3e06-48cc-9d0c-8168c74ae4ea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" Sep 13 20:02:36.973731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9f5m\" (UniqueName: \"kubernetes.io/projected/1f476a24-96a1-4723-ba1d-45bf3df32556-kube-api-access-t9f5m\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.973731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2bx\" (UniqueName: \"kubernetes.io/projected/0940fe8c-22f7-42c0-8323-86799a10fb20-kube-api-access-vq2bx\") pod \"network-check-source-6f8b768455-5bnrs\" (UID: \"0940fe8c-22f7-42c0-8323-86799a10fb20\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" Sep 13 20:02:36.973731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f476a24-96a1-4723-ba1d-45bf3df32556-snapshots\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.973731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973706 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f476a24-96a1-4723-ba1d-45bf3df32556-trusted-ca-bundle\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.973980 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f476a24-96a1-4723-ba1d-45bf3df32556-service-ca-bundle\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.973980 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.973913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f476a24-96a1-4723-ba1d-45bf3df32556-tmp\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.974374 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.974339 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f476a24-96a1-4723-ba1d-45bf3df32556-snapshots\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.974374 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.974361 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f476a24-96a1-4723-ba1d-45bf3df32556-tmp\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.974524 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.974506 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f476a24-96a1-4723-ba1d-45bf3df32556-service-ca-bundle\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.974665 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.974647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f476a24-96a1-4723-ba1d-45bf3df32556-trusted-ca-bundle\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.976140 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.976112 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f476a24-96a1-4723-ba1d-45bf3df32556-serving-cert\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.981499 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.981476 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9f5m\" (UniqueName: \"kubernetes.io/projected/1f476a24-96a1-4723-ba1d-45bf3df32556-kube-api-access-t9f5m\") pod \"insights-operator-5844b856db-shgsc\" (UID: \"1f476a24-96a1-4723-ba1d-45bf3df32556\") " pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:36.981612 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.981533 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6pr2\" (UniqueName: \"kubernetes.io/projected/f9228fd1-3e06-48cc-9d0c-8168c74ae4ea-kube-api-access-n6pr2\") pod \"volume-data-source-validator-86695b7cc7-ps4lh\" (UID: \"f9228fd1-3e06-48cc-9d0c-8168c74ae4ea\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" Sep 13 20:02:36.981612 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.981546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2bx\" (UniqueName: \"kubernetes.io/projected/0940fe8c-22f7-42c0-8323-86799a10fb20-kube-api-access-vq2bx\") pod \"network-check-source-6f8b768455-5bnrs\" (UID: \"0940fe8c-22f7-42c0-8323-86799a10fb20\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" Sep 13 20:02:36.986767 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:36.986739 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:37.069976 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:37.069939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" Sep 13 20:02:37.078104 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:37.078059 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" Sep 13 20:02:37.084065 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:37.084042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-shgsc" Sep 13 20:02:37.341026 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:37.340885 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh"] Sep 13 20:02:37.343542 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:02:37.343515 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9228fd1_3e06_48cc_9d0c_8168c74ae4ea.slice/crio-4c2781e0013763740b15cddce46d8980687de1d5290fb8b6192cfe6a7433f357 WatchSource:0}: Error finding container 4c2781e0013763740b15cddce46d8980687de1d5290fb8b6192cfe6a7433f357: Status 404 returned error can't find the container with id 4c2781e0013763740b15cddce46d8980687de1d5290fb8b6192cfe6a7433f357 Sep 13 20:02:37.350039 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:37.350016 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs"] Sep 13 20:02:37.352803 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:02:37.352780 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0940fe8c_22f7_42c0_8323_86799a10fb20.slice/crio-8c89318e85f0ee8abccd703ef466c32ecd227c475a8a5e04dc8130ae0a8bcb65 WatchSource:0}: Error finding container 8c89318e85f0ee8abccd703ef466c32ecd227c475a8a5e04dc8130ae0a8bcb65: Status 404 returned error can't find the container with id 8c89318e85f0ee8abccd703ef466c32ecd227c475a8a5e04dc8130ae0a8bcb65 Sep 13 20:02:37.367429 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:37.367396 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-shgsc"] Sep 13 20:02:37.370128 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:02:37.370104 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f476a24_96a1_4723_ba1d_45bf3df32556.slice/crio-295c07f184c805618866fbebb3cbdd6615111a352809445f133cd6be0fcb6d41 WatchSource:0}: Error finding container 295c07f184c805618866fbebb3cbdd6615111a352809445f133cd6be0fcb6d41: Status 404 returned error can't find the container with id 295c07f184c805618866fbebb3cbdd6615111a352809445f133cd6be0fcb6d41 Sep 13 20:02:38.316989 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:38.316950 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-shgsc" event={"ID":"1f476a24-96a1-4723-ba1d-45bf3df32556","Type":"ContainerStarted","Data":"295c07f184c805618866fbebb3cbdd6615111a352809445f133cd6be0fcb6d41"} Sep 13 20:02:38.318216 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:38.318174 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" event={"ID":"f9228fd1-3e06-48cc-9d0c-8168c74ae4ea","Type":"ContainerStarted","Data":"4c2781e0013763740b15cddce46d8980687de1d5290fb8b6192cfe6a7433f357"} Sep 13 20:02:38.319731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:38.319701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" event={"ID":"0940fe8c-22f7-42c0-8323-86799a10fb20","Type":"ContainerStarted","Data":"6d2e1e4b0345aed1b5d165b328cfb983f6d37fea8b1f406facfdabf32d0b5b5b"} Sep 13 20:02:38.319848 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:38.319749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" event={"ID":"0940fe8c-22f7-42c0-8323-86799a10fb20","Type":"ContainerStarted","Data":"8c89318e85f0ee8abccd703ef466c32ecd227c475a8a5e04dc8130ae0a8bcb65"} Sep 13 20:02:38.334430 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:38.334382 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5bnrs" podStartSLOduration=2.33436638 podStartE2EDuration="2.33436638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:02:38.333351661 +0000 UTC m=+254.416496576" watchObservedRunningTime="2026-09-13 20:02:38.33436638 +0000 UTC m=+254.417511405" Sep 13 20:02:39.325668 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:39.325623 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" event={"ID":"f9228fd1-3e06-48cc-9d0c-8168c74ae4ea","Type":"ContainerStarted","Data":"b2b2b5a1d1b3f57d6c3af8119beca33ebc389fc376c058312988abaad090edb7"} Sep 13 20:02:39.339169 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:39.339034 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ps4lh" podStartSLOduration=2.115469177 podStartE2EDuration="3.339014355s" totalImagesPullingTime="1.223545178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:02:39.338609972 +0000 UTC m=+255.421754893" watchObservedRunningTime="2026-09-13 20:02:39.339014355 +0000 UTC m=+255.422159270" Sep 13 20:02:39.413067 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:39.413024 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:39.514167 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:39.514138 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:40.330397 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.330357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-shgsc" event={"ID":"1f476a24-96a1-4723-ba1d-45bf3df32556","Type":"ContainerStarted","Data":"5fb3e78672f33075bd6bb57cb93e40f7e71915b38800f209ca98c7e078486d0b"} Sep 13 20:02:40.345002 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.344954 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-shgsc" podStartSLOduration=2.282336166 podStartE2EDuration="4.344940389s" totalImagesPullingTime="2.062604223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:02:40.344583548 +0000 UTC m=+256.427728465" watchObservedRunningTime="2026-09-13 20:02:40.344940389 +0000 UTC m=+256.428085302" Sep 13 20:02:40.791556 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.791519 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:40.800839 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.800815 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn"] Sep 13 20:02:40.803803 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.803787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" Sep 13 20:02:40.805926 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.805865 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 13 20:02:40.806729 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.806708 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nj6wd\"" Sep 13 20:02:40.806830 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.806735 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 13 20:02:40.812596 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.812575 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn"] Sep 13 20:02:40.892309 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.892272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:40.907232 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.907168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjph\" (UniqueName: \"kubernetes.io/projected/b6fd1d72-04ee-4147-a416-571e087fe5ca-kube-api-access-9cjph\") pod \"migrator-8b4d697c5-q2svn\" (UID: \"b6fd1d72-04ee-4147-a416-571e087fe5ca\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" Sep 13 20:02:40.946020 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.945983 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dl98d"] Sep 13 20:02:40.949049 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.949029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:40.951101 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.951082 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 13 20:02:40.951177 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.951082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 13 20:02:40.951439 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.951418 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dlf6m\"" Sep 13 20:02:40.958557 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.958531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dl98d"] Sep 13 20:02:40.974613 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:40.974588 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:41.008541 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.008509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjph\" (UniqueName: \"kubernetes.io/projected/b6fd1d72-04ee-4147-a416-571e087fe5ca-kube-api-access-9cjph\") pod \"migrator-8b4d697c5-q2svn\" (UID: \"b6fd1d72-04ee-4147-a416-571e087fe5ca\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" Sep 13 20:02:41.015744 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.015721 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjph\" (UniqueName: \"kubernetes.io/projected/b6fd1d72-04ee-4147-a416-571e087fe5ca-kube-api-access-9cjph\") pod \"migrator-8b4d697c5-q2svn\" (UID: \"b6fd1d72-04ee-4147-a416-571e087fe5ca\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" Sep 13 20:02:41.109717 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.109629 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.109717 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.109670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82635252-2416-496f-b0ee-50725e625c15-crio-socket\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.109717 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.109702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.109937 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.109732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68k4\" (UniqueName: \"kubernetes.io/projected/82635252-2416-496f-b0ee-50725e625c15-kube-api-access-r68k4\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.109937 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.109797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82635252-2416-496f-b0ee-50725e625c15-data-volume\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.112626 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.112603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" Sep 13 20:02:41.210803 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.210757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.210803 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.210804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82635252-2416-496f-b0ee-50725e625c15-crio-socket\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.210996 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.210873 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82635252-2416-496f-b0ee-50725e625c15-crio-socket\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.210996 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.210876 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 13 20:02:41.210996 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.210916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.210996 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.210945 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:41.710929184 +0000 UTC m=+257.794074081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : configmap "kube-rbac-proxy" not found Sep 13 20:02:41.210996 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.210962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68k4\" (UniqueName: \"kubernetes.io/projected/82635252-2416-496f-b0ee-50725e625c15-kube-api-access-r68k4\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.211154 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.211003 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 13 20:02:41.211154 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.211025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82635252-2416-496f-b0ee-50725e625c15-data-volume\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.211154 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.211043 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:41.711030109 +0000 UTC m=+257.794175005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : secret "insights-runtime-extractor-tls" not found Sep 13 20:02:41.211320 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.211305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82635252-2416-496f-b0ee-50725e625c15-data-volume\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.221147 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.221119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68k4\" (UniqueName: \"kubernetes.io/projected/82635252-2416-496f-b0ee-50725e625c15-kube-api-access-r68k4\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.239290 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.239260 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:41.290946 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.290916 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:41.377342 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.377319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn"] Sep 13 20:02:41.379385 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:02:41.379358 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fd1d72_04ee_4147_a416_571e087fe5ca.slice/crio-8e4a9c6bc286529403275f005f128feaa9c222213a6baa83dd4847edc534f268 WatchSource:0}: Error finding container 8e4a9c6bc286529403275f005f128feaa9c222213a6baa83dd4847edc534f268: Status 404 returned error can't find the container with id 8e4a9c6bc286529403275f005f128feaa9c222213a6baa83dd4847edc534f268 Sep 13 20:02:41.662615 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.662531 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:41.715683 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.715642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.715835 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.715723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:41.715835 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.715787 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 13 20:02:41.715835 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.715816 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 13 20:02:41.715835 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.715837 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:42.715824298 +0000 UTC m=+258.798969196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : configmap "kube-rbac-proxy" not found Sep 13 20:02:41.716011 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:41.715878 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:42.715860936 +0000 UTC m=+258.799005848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : secret "insights-runtime-extractor-tls" not found Sep 13 20:02:41.763598 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:41.763562 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:42.005592 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:42.005494 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:42.105653 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:42.105620 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:42.339833 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:42.339799 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" event={"ID":"b6fd1d72-04ee-4147-a416-571e087fe5ca","Type":"ContainerStarted","Data":"8e4a9c6bc286529403275f005f128feaa9c222213a6baa83dd4847edc534f268"} Sep 13 20:02:42.725945 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:42.725921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:42.726282 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:42.725968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:42.726282 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:42.726047 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 13 20:02:42.726282 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:42.726120 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 13 20:02:42.726282 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:42.726124 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:44.72610412 +0000 UTC m=+260.809249023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : configmap "kube-rbac-proxy" not found Sep 13 20:02:42.726282 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:42.726173 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:44.726162007 +0000 UTC m=+260.809306898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : secret "insights-runtime-extractor-tls" not found Sep 13 20:02:43.344370 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:43.344335 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" event={"ID":"b6fd1d72-04ee-4147-a416-571e087fe5ca","Type":"ContainerStarted","Data":"22ec07b11610761f68f40f6a09a762198d1b213c49bef9a375ea4397660af819"} Sep 13 20:02:43.344565 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:43.344383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" event={"ID":"b6fd1d72-04ee-4147-a416-571e087fe5ca","Type":"ContainerStarted","Data":"e550d85c000ad0d7f379f3a318db5b4a75da3a62009f4ab945b9100c984a8ef5"} Sep 13 20:02:43.358359 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:43.358308 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-q2svn" podStartSLOduration=2.120333917 podStartE2EDuration="3.358293912s" totalImagesPullingTime="1.237959995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:02:43.357036058 +0000 UTC m=+259.440180972" watchObservedRunningTime="2026-09-13 20:02:43.358293912 +0000 UTC m=+259.441438825" Sep 13 20:02:43.419935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:43.419903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:43.470828 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:43.470795 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:43.570618 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:43.570589 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:43.908542 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:43.908511 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x6bb4_f676cc21-4b60-4544-b5b8-1c7d297575c4/dns-node-resolver/0.log" Sep 13 20:02:44.426133 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:44.426092 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:44.741202 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:44.741089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:44.741202 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:44.741140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:44.741432 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:44.741259 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 13 20:02:44.741432 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:44.741274 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 13 20:02:44.741432 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:44.741331 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:48.741316813 +0000 UTC m=+264.824461705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : secret "insights-runtime-extractor-tls" not found Sep 13 20:02:44.741432 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:44.741344 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:48.741338219 +0000 UTC m=+264.824483111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : configmap "kube-rbac-proxy" not found Sep 13 20:02:44.902589 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:44.902562 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xfg7g_9746c0ef-e12e-4b7e-9b92-e3878cc381d9/node-ca/0.log" Sep 13 20:02:45.027339 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.027312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:45.264732 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.264702 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:45.302173 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.302099 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-89mtv_88db34e6-cfe4-4767-9054-ba703ed75960/serve-healthcheck-canary/0.log" Sep 13 20:02:45.348204 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.348153 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:45.449011 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.448981 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:45.549443 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.549412 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:45.674381 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.674304 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:45.778503 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.778468 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:45.903334 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:45.903301 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-q2svn_b6fd1d72-04ee-4147-a416-571e087fe5ca/migrator/0.log" Sep 13 20:02:46.102826 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:46.102798 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-q2svn_b6fd1d72-04ee-4147-a416-571e087fe5ca/graceful-termination/0.log" Sep 13 20:02:46.131015 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:46.130987 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:47.200062 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:47.199992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:47.327560 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:47.327531 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:47.527062 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:47.527032 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:47.723950 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:47.723911 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:47.929097 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:47.929015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:48.051474 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.051434 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" podUID="769ea3eb-d85d-4ccc-85c2-9482ac32f837" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused" Sep 13 20:02:48.122030 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.121992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:48.360543 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.360513 2578 generic.go:356] "Generic (PLEG): container finished" podID="769ea3eb-d85d-4ccc-85c2-9482ac32f837" containerID="4c444a74f13714ef75e82aedbe8de53f1d5814a273a3a331508ea4b71e798c4c" exitCode=1 Sep 13 20:02:48.360998 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.360589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" event={"ID":"769ea3eb-d85d-4ccc-85c2-9482ac32f837","Type":"ContainerDied","Data":"4c444a74f13714ef75e82aedbe8de53f1d5814a273a3a331508ea4b71e798c4c"} Sep 13 20:02:48.360998 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.360932 2578 scope.go:117] "RemoveContainer" containerID="4c444a74f13714ef75e82aedbe8de53f1d5814a273a3a331508ea4b71e798c4c" Sep 13 20:02:48.362100 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.362079 2578 generic.go:356] "Generic (PLEG): container finished" podID="e2793184-45a8-47a3-9fc4-d740a487a5b5" containerID="6844859d6559f5223bb88e745e2e869359468fea8307ad94f3ec6755efa590f4" exitCode=255 Sep 13 20:02:48.362207 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.362141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" event={"ID":"e2793184-45a8-47a3-9fc4-d740a487a5b5","Type":"ContainerDied","Data":"6844859d6559f5223bb88e745e2e869359468fea8307ad94f3ec6755efa590f4"} Sep 13 20:02:48.367925 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.367905 2578 scope.go:117] "RemoveContainer" containerID="6844859d6559f5223bb88e745e2e869359468fea8307ad94f3ec6755efa590f4" Sep 13 20:02:48.776309 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.776272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:48.776309 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.776314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:48.776495 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:48.776410 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 13 20:02:48.776495 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:48.776425 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 13 20:02:48.776495 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:48.776482 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:56.776467358 +0000 UTC m=+272.859612251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : secret "insights-runtime-extractor-tls" not found Sep 13 20:02:48.776619 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:02:48.776512 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm podName:82635252-2416-496f-b0ee-50725e625c15 nodeName:}" failed. No retries permitted until 2026-09-13 20:02:56.776493412 +0000 UTC m=+272.859638308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl98d" (UID: "82635252-2416-496f-b0ee-50725e625c15") : configmap "kube-rbac-proxy" not found Sep 13 20:02:48.925471 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:48.925440 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:49.030754 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.030680 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:49.322510 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.322430 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:49.366664 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.366630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-95fc88884-knnkd" event={"ID":"e2793184-45a8-47a3-9fc4-d740a487a5b5","Type":"ContainerStarted","Data":"f5b05681dfd89a40c01b852fc6e679445570f78eb83a8a252bb23fad567f6003"} Sep 13 20:02:49.368250 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.368228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" event={"ID":"769ea3eb-d85d-4ccc-85c2-9482ac32f837","Type":"ContainerStarted","Data":"a07131372345bba2b06413e87d5ba95ef4d04fb15a1392cfb978a8e51496f032"} Sep 13 20:02:49.368476 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.368461 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:02:49.369011 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.368995 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9f5885746-s7vlr" Sep 13 20:02:49.523391 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.523350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:49.728606 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.728512 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:49.921584 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:49.921540 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:50.126596 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:50.126563 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:51.319805 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:51.319772 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:51.525289 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:51.525258 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:51.626980 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:51.626884 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:51.729103 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:51.729052 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:51.930004 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:51.929920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:52.125096 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:52.125061 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:53.127209 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:53.127151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:53.322600 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:53.322565 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:53.529029 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:53.528997 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:53.721926 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:53.721880 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:53.997071 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:53.996980 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:54.725594 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:54.725564 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:54.947080 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:54.947048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:55.323903 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:55.323857 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:55.527296 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:55.527264 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:55.628706 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:55.628597 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:55.728365 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:55.728323 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:55.829876 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:55.829834 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:55.931699 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:55.931613 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:56.250961 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:56.250871 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:56.838006 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:56.837978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:56.838006 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:56.838017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:56.838500 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:56.838483 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82635252-2416-496f-b0ee-50725e625c15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:56.840250 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:56.840224 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82635252-2416-496f-b0ee-50725e625c15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl98d\" (UID: \"82635252-2416-496f-b0ee-50725e625c15\") " pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:56.858719 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:56.858698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dl98d" Sep 13 20:02:57.122524 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:57.122446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dl98d"] Sep 13 20:02:57.125529 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:02:57.125502 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82635252_2416_496f_b0ee_50725e625c15.slice/crio-7aada36e21d7ec7f8e19f92ac92adb2bb0c752e73b54fa50de354d704d17035a WatchSource:0}: Error finding container 7aada36e21d7ec7f8e19f92ac92adb2bb0c752e73b54fa50de354d704d17035a: Status 404 returned error can't find the container with id 7aada36e21d7ec7f8e19f92ac92adb2bb0c752e73b54fa50de354d704d17035a Sep 13 20:02:57.393111 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:57.393029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl98d" event={"ID":"82635252-2416-496f-b0ee-50725e625c15","Type":"ContainerStarted","Data":"23dc2fd1b0f6e84116385ba1fa011d5589fd75b1f8d3169627768cf039be01e8"} Sep 13 20:02:57.393111 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:57.393080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl98d" event={"ID":"82635252-2416-496f-b0ee-50725e625c15","Type":"ContainerStarted","Data":"7aada36e21d7ec7f8e19f92ac92adb2bb0c752e73b54fa50de354d704d17035a"} Sep 13 20:02:57.831649 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:57.831628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:57.932838 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:57.932808 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:02:58.398221 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:02:58.398171 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl98d" event={"ID":"82635252-2416-496f-b0ee-50725e625c15","Type":"ContainerStarted","Data":"dc7499d5c026aaf58b1a023d6fd20e6df59fb32b5434720cbf200aaab7cfab1a"} Sep 13 20:03:00.406839 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:00.406804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl98d" event={"ID":"82635252-2416-496f-b0ee-50725e625c15","Type":"ContainerStarted","Data":"94f920426d37c83282833ba7631557ef48166a3c09ea2e1f1a5aa98ab424e0ab"} Sep 13 20:03:00.422127 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:00.422077 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dl98d" podStartSLOduration=18.164645758 podStartE2EDuration="20.422061503s" totalImagesPullingTime="2.257415745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 20:02:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:00.421233841 +0000 UTC m=+276.504378756" watchObservedRunningTime="2026-09-13 20:03:00.422061503 +0000 UTC m=+276.505206416" Sep 13 20:03:02.242127 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:02.242100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:02.342693 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:02.342661 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:02.537228 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:02.537201 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:02.924160 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:02.924077 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:03.327981 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:03.327952 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:03.744611 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:03.744538 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:03.928479 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:03.928448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:04.029111 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:04.029086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:04.128944 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:04.128906 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:04.230177 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:04.230147 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:05.911627 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:05.911594 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-plznc"] Sep 13 20:03:05.914733 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:05.914718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-plznc" Sep 13 20:03:05.917027 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:05.917006 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b8fmr\"" Sep 13 20:03:05.917152 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:05.917050 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 13 20:03:05.917152 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:05.917049 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 13 20:03:05.937687 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:05.937667 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-plznc"] Sep 13 20:03:06.001037 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.001008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dp8s\" (UniqueName: \"kubernetes.io/projected/9ee869a4-5438-45db-b9e4-ee6dfd603156-kube-api-access-5dp8s\") pod \"downloads-76f99584f4-plznc\" (UID: \"9ee869a4-5438-45db-b9e4-ee6dfd603156\") " pod="openshift-console/downloads-76f99584f4-plznc" Sep 13 20:03:06.011664 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.011638 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54ccd6f8b8-b5z66"] Sep 13 20:03:06.014492 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.014478 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.017058 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.017039 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fshcc\"" Sep 13 20:03:06.017118 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.017062 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 13 20:03:06.017118 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.017101 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 13 20:03:06.018558 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.018542 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 13 20:03:06.021761 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.021742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 13 20:03:06.028407 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.028386 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54ccd6f8b8-b5z66"] Sep 13 20:03:06.046016 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.045990 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:06.089773 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:03:06.089729 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jcz7f" podUID="fd881c97-f6a3-46b3-922c-451a18fdfafd" Sep 13 20:03:06.102096 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd215c20-90f9-4732-a877-544e958b54be-trusted-ca\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.102229 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-registry-tls\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.102229 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd215c20-90f9-4732-a877-544e958b54be-image-registry-private-configuration\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.102346 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-bound-sa-token\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.102346 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd215c20-90f9-4732-a877-544e958b54be-ca-trust-extracted\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.102346 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd215c20-90f9-4732-a877-544e958b54be-installation-pull-secrets\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.102346 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd215c20-90f9-4732-a877-544e958b54be-registry-certificates\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.102513 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dp8s\" (UniqueName: \"kubernetes.io/projected/9ee869a4-5438-45db-b9e4-ee6dfd603156-kube-api-access-5dp8s\") pod \"downloads-76f99584f4-plznc\" (UID: \"9ee869a4-5438-45db-b9e4-ee6dfd603156\") " pod="openshift-console/downloads-76f99584f4-plznc" Sep 13 20:03:06.102513 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.102377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq54c\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-kube-api-access-cq54c\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.111069 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.111043 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dp8s\" (UniqueName: \"kubernetes.io/projected/9ee869a4-5438-45db-b9e4-ee6dfd603156-kube-api-access-5dp8s\") pod \"downloads-76f99584f4-plznc\" (UID: \"9ee869a4-5438-45db-b9e4-ee6dfd603156\") " pod="openshift-console/downloads-76f99584f4-plznc" Sep 13 20:03:06.145569 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.145543 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:06.203527 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-bound-sa-token\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203527 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd215c20-90f9-4732-a877-544e958b54be-ca-trust-extracted\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203696 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd215c20-90f9-4732-a877-544e958b54be-installation-pull-secrets\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203696 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd215c20-90f9-4732-a877-544e958b54be-registry-certificates\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203696 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq54c\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-kube-api-access-cq54c\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203696 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd215c20-90f9-4732-a877-544e958b54be-trusted-ca\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203912 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-registry-tls\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203912 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd215c20-90f9-4732-a877-544e958b54be-image-registry-private-configuration\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.203912 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.203868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd215c20-90f9-4732-a877-544e958b54be-ca-trust-extracted\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.204463 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.204440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd215c20-90f9-4732-a877-544e958b54be-registry-certificates\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.204693 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.204667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd215c20-90f9-4732-a877-544e958b54be-trusted-ca\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.205820 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.205793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd215c20-90f9-4732-a877-544e958b54be-installation-pull-secrets\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.206027 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.206011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-registry-tls\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.206097 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.206055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd215c20-90f9-4732-a877-544e958b54be-image-registry-private-configuration\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.211597 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.211572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-bound-sa-token\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.211675 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.211594 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq54c\" (UniqueName: \"kubernetes.io/projected/cd215c20-90f9-4732-a877-544e958b54be-kube-api-access-cq54c\") pod \"image-registry-54ccd6f8b8-b5z66\" (UID: \"cd215c20-90f9-4732-a877-544e958b54be\") " pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.223400 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.223380 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-plznc" Sep 13 20:03:06.323118 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.323088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:06.427601 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.427568 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jcz7f" Sep 13 20:03:06.489277 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.489208 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-plznc"] Sep 13 20:03:06.491784 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:03:06.491760 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee869a4_5438_45db_b9e4_ee6dfd603156.slice/crio-a9a3c13a1c263ee1d9100bef14999e23812796eb8d0aa257e9d0f82fdcfb4828 WatchSource:0}: Error finding container a9a3c13a1c263ee1d9100bef14999e23812796eb8d0aa257e9d0f82fdcfb4828: Status 404 returned error can't find the container with id a9a3c13a1c263ee1d9100bef14999e23812796eb8d0aa257e9d0f82fdcfb4828 Sep 13 20:03:06.589619 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:06.589599 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54ccd6f8b8-b5z66"] Sep 13 20:03:06.591674 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:03:06.591654 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd215c20_90f9_4732_a877_544e958b54be.slice/crio-489c87f58a19a73503e5397bea68b9c0eb0de347bf4bc2ebbc0d7236f68c4b53 WatchSource:0}: Error finding container 489c87f58a19a73503e5397bea68b9c0eb0de347bf4bc2ebbc0d7236f68c4b53: Status 404 returned error can't find the container with id 489c87f58a19a73503e5397bea68b9c0eb0de347bf4bc2ebbc0d7236f68c4b53 Sep 13 20:03:07.432416 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:07.432379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-plznc" event={"ID":"9ee869a4-5438-45db-b9e4-ee6dfd603156","Type":"ContainerStarted","Data":"a9a3c13a1c263ee1d9100bef14999e23812796eb8d0aa257e9d0f82fdcfb4828"} Sep 13 20:03:07.434062 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:07.434038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" event={"ID":"cd215c20-90f9-4732-a877-544e958b54be","Type":"ContainerStarted","Data":"3f49b6b02592e497617f3acbb43cb7fc72fa21a48cf52d9f61af4ec3bc4e18c6"} Sep 13 20:03:07.434216 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:07.434076 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" event={"ID":"cd215c20-90f9-4732-a877-544e958b54be","Type":"ContainerStarted","Data":"489c87f58a19a73503e5397bea68b9c0eb0de347bf4bc2ebbc0d7236f68c4b53"} Sep 13 20:03:07.434216 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:07.434122 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:07.452060 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:07.452018 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" podStartSLOduration=2.452005183 podStartE2EDuration="2.452005183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:07.450642402 +0000 UTC m=+283.533787339" watchObservedRunningTime="2026-09-13 20:03:07.452005183 +0000 UTC m=+283.535150096" Sep 13 20:03:08.895123 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:08.895088 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:08.952812 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:08.952778 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:09.215783 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:09.215696 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:09.231727 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:09.231697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 20:03:09.234316 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:09.234296 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd881c97-f6a3-46b3-922c-451a18fdfafd-metrics-tls\") pod \"dns-default-jcz7f\" (UID: \"fd881c97-f6a3-46b3-922c-451a18fdfafd\") " pod="openshift-dns/dns-default-jcz7f" Sep 13 20:03:09.432087 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:09.432046 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pm8n4\"" Sep 13 20:03:09.439123 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:09.439094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jcz7f" Sep 13 20:03:09.717246 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:09.717217 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jcz7f"] Sep 13 20:03:09.721284 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:03:09.721239 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd881c97_f6a3_46b3_922c_451a18fdfafd.slice/crio-775702e878d24681faa0eeb83f63e2a8745a3706635d4f20525d1a1453a3e694 WatchSource:0}: Error finding container 775702e878d24681faa0eeb83f63e2a8745a3706635d4f20525d1a1453a3e694: Status 404 returned error can't find the container with id 775702e878d24681faa0eeb83f63e2a8745a3706635d4f20525d1a1453a3e694 Sep 13 20:03:10.447774 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:10.447732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jcz7f" event={"ID":"fd881c97-f6a3-46b3-922c-451a18fdfafd","Type":"ContainerStarted","Data":"775702e878d24681faa0eeb83f63e2a8745a3706635d4f20525d1a1453a3e694"} Sep 13 20:03:11.129828 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:11.129792 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:12.457877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:12.457831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jcz7f" event={"ID":"fd881c97-f6a3-46b3-922c-451a18fdfafd","Type":"ContainerStarted","Data":"6fa7768436feed20abb5266899520b8d39c68ab0b1c1a79bf4cfc1d7909d4684"} Sep 13 20:03:12.458378 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:12.457890 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jcz7f" event={"ID":"fd881c97-f6a3-46b3-922c-451a18fdfafd","Type":"ContainerStarted","Data":"c460fb37b7e67517292f4d42199a79e4aee8706af61b3ca9d5778323d12e2a63"} Sep 13 20:03:12.458378 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:12.457939 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jcz7f" Sep 13 20:03:12.474232 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:12.474151 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jcz7f" podStartSLOduration=251.597548765 podStartE2EDuration="4m13.474129662s" totalImagesPullingTime="1.876580897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 19:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:12.472532965 +0000 UTC m=+288.555677889" watchObservedRunningTime="2026-09-13 20:03:12.474129662 +0000 UTC m=+288.557274580" Sep 13 20:03:13.018530 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.018496 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.125579 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.125548 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.240318 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.240280 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.269697 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.269622 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb"] Sep 13 20:03:13.275229 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.275196 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vvj6j"] Sep 13 20:03:13.275404 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.275306 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.277907 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.277875 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z5f47\"" Sep 13 20:03:13.278051 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.277931 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 13 20:03:13.278815 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.278785 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 13 20:03:13.279038 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.278864 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 13 20:03:13.279038 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.278874 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 13 20:03:13.279168 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.279104 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 13 20:03:13.279244 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.279175 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 13 20:03:13.279878 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.279855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.284029 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.283514 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 13 20:03:13.284029 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.283986 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 13 20:03:13.284492 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.284075 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 13 20:03:13.284492 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.284289 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qnhnl\"" Sep 13 20:03:13.289269 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.289147 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb"] Sep 13 20:03:13.315902 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.315868 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.365651 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-tls\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.365823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-wtmp\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.365823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.365823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365734 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.365823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2dh\" (UniqueName: \"kubernetes.io/projected/f563bbe7-aa7b-4a12-967a-e01253aa7d72-kube-api-access-6v2dh\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.365823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-root\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.365823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlhr\" (UniqueName: \"kubernetes.io/projected/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-api-access-kwlhr\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.366074 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.366074 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-sys\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.366074 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f563bbe7-aa7b-4a12-967a-e01253aa7d72-metrics-client-ca\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.366074 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.365995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.366074 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.366042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.366289 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.366091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.366289 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.366143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-textfile\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.366289 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.366175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.381145 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.381111 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.466901 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.466869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.466922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-sys\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.466987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-sys\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f563bbe7-aa7b-4a12-967a-e01253aa7d72-metrics-client-ca\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-textfile\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-tls\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-wtmp\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2dh\" (UniqueName: \"kubernetes.io/projected/f563bbe7-aa7b-4a12-967a-e01253aa7d72-kube-api-access-6v2dh\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-root\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.467411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlhr\" (UniqueName: \"kubernetes.io/projected/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-api-access-kwlhr\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.468106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467649 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f563bbe7-aa7b-4a12-967a-e01253aa7d72-metrics-client-ca\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.468106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467853 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-wtmp\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.468106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.467881 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.468309 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.468143 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f563bbe7-aa7b-4a12-967a-e01253aa7d72-root\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.468459 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.468415 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.468590 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.468482 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-textfile\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.468590 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.468490 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.468965 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.468906 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.470784 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.470762 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.470896 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.470766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.471229 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.471208 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-tls\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.471340 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.471319 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f563bbe7-aa7b-4a12-967a-e01253aa7d72-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.475244 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.475219 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2dh\" (UniqueName: \"kubernetes.io/projected/f563bbe7-aa7b-4a12-967a-e01253aa7d72-kube-api-access-6v2dh\") pod \"node-exporter-vvj6j\" (UID: \"f563bbe7-aa7b-4a12-967a-e01253aa7d72\") " pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.475517 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.475501 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlhr\" (UniqueName: \"kubernetes.io/projected/5a4f4bcd-70c7-4e0d-842d-be06c2aea763-kube-api-access-kwlhr\") pod \"kube-state-metrics-59fbdd77d-krnsb\" (UID: \"5a4f4bcd-70c7-4e0d-842d-be06c2aea763\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.483652 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.483616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.589156 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.589073 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" Sep 13 20:03:13.596294 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.596258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvj6j" Sep 13 20:03:13.614695 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.614660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.656226 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:03:13.656120 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf563bbe7_aa7b_4a12_967a_e01253aa7d72.slice/crio-a9a4dbb8af65f179d348b1be48aada67ec3bc87ae2e4be1ce6cb6be17c603fbd WatchSource:0}: Error finding container a9a4dbb8af65f179d348b1be48aada67ec3bc87ae2e4be1ce6cb6be17c603fbd: Status 404 returned error can't find the container with id a9a4dbb8af65f179d348b1be48aada67ec3bc87ae2e4be1ce6cb6be17c603fbd Sep 13 20:03:13.715818 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.715785 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:13.907003 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:13.906930 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb"] Sep 13 20:03:13.910208 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:03:13.910161 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4f4bcd_70c7_4e0d_842d_be06c2aea763.slice/crio-32067c29b8fa594bb0c4343b02665842e17d47b91232c6facf6bff9b66beabe3 WatchSource:0}: Error finding container 32067c29b8fa594bb0c4343b02665842e17d47b91232c6facf6bff9b66beabe3: Status 404 returned error can't find the container with id 32067c29b8fa594bb0c4343b02665842e17d47b91232c6facf6bff9b66beabe3 Sep 13 20:03:14.104419 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:14.104377 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:14.209982 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:14.209862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:14.305635 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:14.305568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:14.372427 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:14.372393 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:14.469102 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:14.469018 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" event={"ID":"5a4f4bcd-70c7-4e0d-842d-be06c2aea763","Type":"ContainerStarted","Data":"32067c29b8fa594bb0c4343b02665842e17d47b91232c6facf6bff9b66beabe3"} Sep 13 20:03:14.470602 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:14.470574 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvj6j" event={"ID":"f563bbe7-aa7b-4a12-967a-e01253aa7d72","Type":"ContainerStarted","Data":"a9a4dbb8af65f179d348b1be48aada67ec3bc87ae2e4be1ce6cb6be17c603fbd"} Sep 13 20:03:14.768109 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:14.768055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:15.252617 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.252537 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7484c56547-qzlfj"] Sep 13 20:03:15.266792 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.266757 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.266961 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.266922 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7484c56547-qzlfj"] Sep 13 20:03:15.270067 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.269630 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 13 20:03:15.270067 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.269903 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 13 20:03:15.270268 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.270115 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 13 20:03:15.270268 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.270224 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 13 20:03:15.270402 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.270345 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x7b99\"" Sep 13 20:03:15.270475 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.270444 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 13 20:03:15.270526 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.270122 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ajgu31992ml2\"" Sep 13 20:03:15.385329 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.385329 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.385573 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlw7v\" (UniqueName: \"kubernetes.io/projected/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-kube-api-access-jlw7v\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.385573 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-grpc-tls\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.385573 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385462 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-tls\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.385573 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.385573 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.385763 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.385604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-metrics-client-ca\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.485998 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.485957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlw7v\" (UniqueName: \"kubernetes.io/projected/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-kube-api-access-jlw7v\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.486491 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.486015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-grpc-tls\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.486491 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.486061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-tls\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.486491 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.486103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.486491 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.486129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.486491 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.486165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-metrics-client-ca\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.486491 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.486246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.486491 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.486270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.487583 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.487553 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-metrics-client-ca\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.489312 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.489283 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.489829 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.489804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.489912 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.489854 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.490205 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.490162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-tls\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.490205 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.490175 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-grpc-tls\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.490803 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.490780 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.495941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.495910 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlw7v\" (UniqueName: \"kubernetes.io/projected/fd3fc4b5-3397-4372-bc27-57dcf2f23f94-kube-api-access-jlw7v\") pod \"thanos-querier-7484c56547-qzlfj\" (UID: \"fd3fc4b5-3397-4372-bc27-57dcf2f23f94\") " pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:15.580430 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:15.580396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:16.066590 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.066446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7484c56547-qzlfj"] Sep 13 20:03:16.165683 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.165651 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:16.480836 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.480743 2578 generic.go:356] "Generic (PLEG): container finished" podID="f563bbe7-aa7b-4a12-967a-e01253aa7d72" containerID="f8274b411ef4cc317f6a2167bc5f06f0d0e8153ef2a9777be15ae833740927bd" exitCode=0 Sep 13 20:03:16.481002 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.480831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvj6j" event={"ID":"f563bbe7-aa7b-4a12-967a-e01253aa7d72","Type":"ContainerDied","Data":"f8274b411ef4cc317f6a2167bc5f06f0d0e8153ef2a9777be15ae833740927bd"} Sep 13 20:03:16.482249 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.482214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" event={"ID":"fd3fc4b5-3397-4372-bc27-57dcf2f23f94","Type":"ContainerStarted","Data":"a5926b9c79f0bac38a0ee6eae0e7fc823d8bd4309bcede49e9ec612761abcd16"} Sep 13 20:03:16.484808 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.484788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" event={"ID":"5a4f4bcd-70c7-4e0d-842d-be06c2aea763","Type":"ContainerStarted","Data":"cae9753c036a8eee900f90376161b9033037218c1fed56bc46307acb742cc836"} Sep 13 20:03:16.484927 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.484828 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" event={"ID":"5a4f4bcd-70c7-4e0d-842d-be06c2aea763","Type":"ContainerStarted","Data":"8fd88cc50a9a6b2b8d471c229d91e13d06c014d73b33f96a241c34055ed09976"} Sep 13 20:03:16.484927 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.484849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" event={"ID":"5a4f4bcd-70c7-4e0d-842d-be06c2aea763","Type":"ContainerStarted","Data":"a7f8338b2f8b6aebe62c5f1162854c12d776b823ff1274a96f93be073ceef416"} Sep 13 20:03:16.511771 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:16.511718 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-krnsb" podStartSLOduration=1.639831203 podStartE2EDuration="3.511697735s" totalImagesPullingTime="1.871866532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:16.510959182 +0000 UTC m=+292.594104109" watchObservedRunningTime="2026-09-13 20:03:16.511697735 +0000 UTC m=+292.594842671" Sep 13 20:03:17.464285 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:17.464240 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:18.039730 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.039689 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:18.050819 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.050786 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2"] Sep 13 20:03:18.064321 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.064269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2"] Sep 13 20:03:18.064513 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.064393 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.066843 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.066818 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 13 20:03:18.066989 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.066871 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8vxgv\"" Sep 13 20:03:18.112621 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.112577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e38f87e0-48bf-427b-85da-c36d319cccb8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-btvc2\" (UID: \"e38f87e0-48bf-427b-85da-c36d319cccb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.112893 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.112667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8wk\" (UniqueName: \"kubernetes.io/projected/e38f87e0-48bf-427b-85da-c36d319cccb8-kube-api-access-dh8wk\") pod \"monitoring-plugin-5d85778b9d-btvc2\" (UID: \"e38f87e0-48bf-427b-85da-c36d319cccb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.136334 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.136294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:18.190686 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.190652 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:18.213323 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.213287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e38f87e0-48bf-427b-85da-c36d319cccb8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-btvc2\" (UID: \"e38f87e0-48bf-427b-85da-c36d319cccb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.213505 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.213365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8wk\" (UniqueName: \"kubernetes.io/projected/e38f87e0-48bf-427b-85da-c36d319cccb8-kube-api-access-dh8wk\") pod \"monitoring-plugin-5d85778b9d-btvc2\" (UID: \"e38f87e0-48bf-427b-85da-c36d319cccb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.215973 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.215946 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e38f87e0-48bf-427b-85da-c36d319cccb8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-btvc2\" (UID: \"e38f87e0-48bf-427b-85da-c36d319cccb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.221033 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.220980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8wk\" (UniqueName: \"kubernetes.io/projected/e38f87e0-48bf-427b-85da-c36d319cccb8-kube-api-access-dh8wk\") pod \"monitoring-plugin-5d85778b9d-btvc2\" (UID: \"e38f87e0-48bf-427b-85da-c36d319cccb8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.377112 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:18.377024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:18.414947 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:03:18.414898 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xk2rl" podUID="2473e3dd-c0cf-4159-8b9d-230c808ea682" Sep 13 20:03:19.445675 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:19.445643 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:19.510673 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:19.510639 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:19.609493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:19.609455 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:19.709730 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:19.709642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:19.811329 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:19.811292 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:20.151364 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.151332 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:20.159394 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.159362 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fbdc6748b-txgfj"] Sep 13 20:03:20.162897 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.162877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.164960 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.164937 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 13 20:03:20.165935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.165896 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 13 20:03:20.166038 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.166024 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 13 20:03:20.166102 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.166042 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 13 20:03:20.166284 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.166265 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vwxwj\"" Sep 13 20:03:20.166357 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.166333 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 13 20:03:20.172207 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.172162 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbdc6748b-txgfj"] Sep 13 20:03:20.176109 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.176083 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 13 20:03:20.234198 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.234156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-serving-cert\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.234371 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.234223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-oauth-serving-cert\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.234371 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.234247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-oauth-config\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.234371 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.234270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtm5\" (UniqueName: \"kubernetes.io/projected/24788071-b196-4971-bf07-f0166dda2ce8-kube-api-access-pdtm5\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.234534 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.234402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-service-ca\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.234534 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.234453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-trusted-ca-bundle\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.234534 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.234519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-console-config\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.250283 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.250244 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:20.335910 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.335865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-console-config\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.335953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-serving-cert\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.335976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-oauth-serving-cert\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.335999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-oauth-config\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.336022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtm5\" (UniqueName: \"kubernetes.io/projected/24788071-b196-4971-bf07-f0166dda2ce8-kube-api-access-pdtm5\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336328 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.336197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-service-ca\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336328 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.336255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-trusted-ca-bundle\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336719 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.336690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-console-config\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.336821 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.336718 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-oauth-serving-cert\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.337002 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.336967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-service-ca\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.337130 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.337070 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-trusted-ca-bundle\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.339124 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.339099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-oauth-config\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.339250 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.339202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-serving-cert\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.343106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.343083 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtm5\" (UniqueName: \"kubernetes.io/projected/24788071-b196-4971-bf07-f0166dda2ce8-kube-api-access-pdtm5\") pod \"console-fbdc6748b-txgfj\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:20.477104 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:20.477027 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:22.463803 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:22.463766 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jcz7f" Sep 13 20:03:23.480374 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.480284 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2"] Sep 13 20:03:23.483627 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:03:23.483586 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38f87e0_48bf_427b_85da_c36d319cccb8.slice/crio-cb624763259887e5c3a3db61078a67465dd5e24d4e9f516a2bec12b949c9df83 WatchSource:0}: Error finding container cb624763259887e5c3a3db61078a67465dd5e24d4e9f516a2bec12b949c9df83: Status 404 returned error can't find the container with id cb624763259887e5c3a3db61078a67465dd5e24d4e9f516a2bec12b949c9df83 Sep 13 20:03:23.486201 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.486158 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbdc6748b-txgfj"] Sep 13 20:03:23.489917 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:03:23.489887 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24788071_b196_4971_bf07_f0166dda2ce8.slice/crio-d132b773d5e629c2e9293139d2f2a1359eb3bf6ad1ae2807d422e8071207ccbe WatchSource:0}: Error finding container d132b773d5e629c2e9293139d2f2a1359eb3bf6ad1ae2807d422e8071207ccbe: Status 404 returned error can't find the container with id d132b773d5e629c2e9293139d2f2a1359eb3bf6ad1ae2807d422e8071207ccbe Sep 13 20:03:23.516160 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.516117 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" event={"ID":"e38f87e0-48bf-427b-85da-c36d319cccb8","Type":"ContainerStarted","Data":"cb624763259887e5c3a3db61078a67465dd5e24d4e9f516a2bec12b949c9df83"} Sep 13 20:03:23.518259 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.518227 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-plznc" event={"ID":"9ee869a4-5438-45db-b9e4-ee6dfd603156","Type":"ContainerStarted","Data":"e0ee2fbf25df948148e102c0e41cce01c97a5ead8d2b19bed80717a506ecad50"} Sep 13 20:03:23.519109 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.518628 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-plznc" Sep 13 20:03:23.520610 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.520333 2578 patch_prober.go:28] interesting pod/downloads-76f99584f4-plznc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" start-of-body= Sep 13 20:03:23.520610 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.520386 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-plznc" podUID="9ee869a4-5438-45db-b9e4-ee6dfd603156" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" Sep 13 20:03:23.521904 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.521816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvj6j" event={"ID":"f563bbe7-aa7b-4a12-967a-e01253aa7d72","Type":"ContainerStarted","Data":"9e8ec672ee23d0328cd7b3d1625e76ab5123843f54d50875482fdc285e2994ff"} Sep 13 20:03:23.521904 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.521864 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvj6j" event={"ID":"f563bbe7-aa7b-4a12-967a-e01253aa7d72","Type":"ContainerStarted","Data":"8779fc194c0e78b9ae4bfc10fd85a7ea74e6c919d9893ac4ed6243506956eb5a"} Sep 13 20:03:23.523571 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.523523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdc6748b-txgfj" event={"ID":"24788071-b196-4971-bf07-f0166dda2ce8","Type":"ContainerStarted","Data":"d132b773d5e629c2e9293139d2f2a1359eb3bf6ad1ae2807d422e8071207ccbe"} Sep 13 20:03:23.536943 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.536883 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-plznc" podStartSLOduration=1.714428767 podStartE2EDuration="18.536862964s" totalImagesPullingTime="16.822434197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:23.536397544 +0000 UTC m=+299.619542486" watchObservedRunningTime="2026-09-13 20:03:23.536862964 +0000 UTC m=+299.620007879" Sep 13 20:03:23.555429 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:23.555102 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vvj6j" podStartSLOduration=8.316138294 podStartE2EDuration="10.555078822s" totalImagesPullingTime="2.123866326s" totalInitContainerRuntime="115.074202ms" isStatefulPod=true podCreationTimestamp="2026-09-13 20:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:23.553016363 +0000 UTC m=+299.636161279" watchObservedRunningTime="2026-09-13 20:03:23.555078822 +0000 UTC m=+299.638223763" Sep 13 20:03:24.136955 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:24.136880 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:24.237109 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:24.237021 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:24.408089 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:24.406719 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 13 20:03:24.544443 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:24.544260 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-plznc" Sep 13 20:03:24.570585 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:24.570530 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:24.627768 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:24.627592 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:27.105555 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:27.105453 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:27.205764 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:27.205684 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:27.832346 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:27.832300 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:27.891493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:27.891456 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:28.426258 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.426175 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:28.443578 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.443547 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54ccd6f8b8-b5z66" Sep 13 20:03:28.496953 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.496828 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:28.551883 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.551826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" event={"ID":"fd3fc4b5-3397-4372-bc27-57dcf2f23f94","Type":"ContainerStarted","Data":"e7b6c2cda7f0f5e803fe5d3bbd85c9855658c33b216ec6ae5c6f7be519c74ce4"} Sep 13 20:03:28.552055 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.551913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" event={"ID":"fd3fc4b5-3397-4372-bc27-57dcf2f23f94","Type":"ContainerStarted","Data":"dbcd0fea37f1fe02041ec93a325e596fb8d11bfd19638c4e1c1987e090a2884a"} Sep 13 20:03:28.553851 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.553822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" event={"ID":"e38f87e0-48bf-427b-85da-c36d319cccb8","Type":"ContainerStarted","Data":"ce1349a7f5bc77a5cb5464c65d328388724d5d59a67b9f803f15e7b44c4b91fb"} Sep 13 20:03:28.554505 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.554485 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:28.559205 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.559078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdc6748b-txgfj" event={"ID":"24788071-b196-4971-bf07-f0166dda2ce8","Type":"ContainerStarted","Data":"945a8c838ac8b7fa4cb6306d05529b077aac206cd331495a898a1024388154c5"} Sep 13 20:03:28.561020 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.560976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" Sep 13 20:03:28.565156 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.565116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:28.573045 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.572976 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-btvc2" podStartSLOduration=5.828999631 podStartE2EDuration="10.572955887s" totalImagesPullingTime="4.743956256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:28.571941477 +0000 UTC m=+304.655086392" watchObservedRunningTime="2026-09-13 20:03:28.572955887 +0000 UTC m=+304.656100817" Sep 13 20:03:28.602855 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.602690 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fbdc6748b-txgfj" podStartSLOduration=3.857676313 podStartE2EDuration="8.602672116s" totalImagesPullingTime="4.744995803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:28.601854263 +0000 UTC m=+304.684999175" watchObservedRunningTime="2026-09-13 20:03:28.602672116 +0000 UTC m=+304.685817032" Sep 13 20:03:28.637777 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.637727 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 13 20:03:28.665784 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:28.665729 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:29.565881 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:29.565833 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" event={"ID":"fd3fc4b5-3397-4372-bc27-57dcf2f23f94","Type":"ContainerStarted","Data":"08de3ebc48350a3ce3e6ff61ae4bd581b3becc4af242ea0d4e5f8cf7ece59989"} Sep 13 20:03:30.478125 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.478036 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:30.478125 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.478105 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:30.483892 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.483858 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:30.573856 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.573819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" event={"ID":"fd3fc4b5-3397-4372-bc27-57dcf2f23f94","Type":"ContainerStarted","Data":"d185bd86ea6cd69cf982a3a750875f0bd33db6dd9f668952d3f8d6be57066f72"} Sep 13 20:03:30.574349 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.573894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" event={"ID":"fd3fc4b5-3397-4372-bc27-57dcf2f23f94","Type":"ContainerStarted","Data":"a7c1bf3d82103950bea174e1317ce6e8e06cdec91db944b8f8ff9d13553d77e5"} Sep 13 20:03:30.574349 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.573946 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" event={"ID":"fd3fc4b5-3397-4372-bc27-57dcf2f23f94","Type":"ContainerStarted","Data":"22c7d621b722cc104fa0fb7f66cc039db335360d2a83d794cc104b682ee69842"} Sep 13 20:03:30.578534 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.578504 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:03:30.594891 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:30.594842 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" podStartSLOduration=2.130756361 podStartE2EDuration="15.594821129s" totalImagesPullingTime="13.464064768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:03:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:03:30.593168354 +0000 UTC m=+306.676313270" watchObservedRunningTime="2026-09-13 20:03:30.594821129 +0000 UTC m=+306.677966043" Sep 13 20:03:31.578979 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:31.578947 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:33.352959 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:33.352923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:33.404361 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:33.404328 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:33.414465 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:33.414444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 20:03:37.151198 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:37.151147 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:37.201176 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:37.201144 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:37.424032 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:37.423958 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:37.473530 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:37.473500 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:37.588861 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:37.588839 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7484c56547-qzlfj" Sep 13 20:03:42.831815 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:42.831780 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:42.886706 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:42.886678 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:43.553886 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:43.553856 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:03:43.606058 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:03:43.606027 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:00.688108 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:00.688075 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f476a24-96a1-4723-ba1d-45bf3df32556" containerID="5fb3e78672f33075bd6bb57cb93e40f7e71915b38800f209ca98c7e078486d0b" exitCode=0 Sep 13 20:04:00.688486 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:00.688155 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-shgsc" event={"ID":"1f476a24-96a1-4723-ba1d-45bf3df32556","Type":"ContainerDied","Data":"5fb3e78672f33075bd6bb57cb93e40f7e71915b38800f209ca98c7e078486d0b"} Sep 13 20:04:00.688544 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:00.688530 2578 scope.go:117] "RemoveContainer" containerID="5fb3e78672f33075bd6bb57cb93e40f7e71915b38800f209ca98c7e078486d0b" Sep 13 20:04:01.693853 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:01.693817 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-shgsc" event={"ID":"1f476a24-96a1-4723-ba1d-45bf3df32556","Type":"ContainerStarted","Data":"74abcc77c9893a11fce8e556ac1d83bcff64235996d1b137a90025abacddebe2"} Sep 13 20:04:29.824683 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:29.824604 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:29.882290 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:29.882254 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:32.421890 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:32.421859 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:32.689626 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:32.689532 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:35.752964 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:35.752931 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:35.853523 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:35.853489 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:38.215349 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:38.215314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 20:04:38.217499 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:38.217479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2473e3dd-c0cf-4159-8b9d-230c808ea682-metrics-certs\") pod \"network-metrics-daemon-xk2rl\" (UID: \"2473e3dd-c0cf-4159-8b9d-230c808ea682\") " pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 20:04:38.517032 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:38.517005 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hvn9g\"" Sep 13 20:04:38.525744 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:38.525721 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2rl" Sep 13 20:04:38.812337 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:38.812136 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk2rl"] Sep 13 20:04:38.815085 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:04:38.815056 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2473e3dd_c0cf_4159_8b9d_230c808ea682.slice/crio-fbddfa3e1dcace001ebe1374cde0da1c7ecbb5d79cc32c204406e3f987d830a4 WatchSource:0}: Error finding container fbddfa3e1dcace001ebe1374cde0da1c7ecbb5d79cc32c204406e3f987d830a4: Status 404 returned error can't find the container with id fbddfa3e1dcace001ebe1374cde0da1c7ecbb5d79cc32c204406e3f987d830a4 Sep 13 20:04:38.838558 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:38.838526 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk2rl" event={"ID":"2473e3dd-c0cf-4159-8b9d-230c808ea682","Type":"ContainerStarted","Data":"fbddfa3e1dcace001ebe1374cde0da1c7ecbb5d79cc32c204406e3f987d830a4"} Sep 13 20:04:39.843198 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:39.843160 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk2rl" event={"ID":"2473e3dd-c0cf-4159-8b9d-230c808ea682","Type":"ContainerStarted","Data":"b3445e689d54a7e1dc0a1145145b68edd0f079eb7a3847e34bd677955009e4ab"} Sep 13 20:04:40.848655 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:40.848617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk2rl" event={"ID":"2473e3dd-c0cf-4159-8b9d-230c808ea682","Type":"ContainerStarted","Data":"4d707607f54f11c2becbb84c78e16da9675334f317d44a3727dea98458d44b5f"} Sep 13 20:04:40.863065 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:40.863013 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xk2rl" podStartSLOduration=374.999839365 podStartE2EDuration="6m15.862995934s" totalImagesPullingTime="863.156569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 19:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:04:40.86229536 +0000 UTC m=+376.945440286" watchObservedRunningTime="2026-09-13 20:04:40.862995934 +0000 UTC m=+376.946140826" Sep 13 20:04:40.943477 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:40.943449 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:41.006698 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:41.006668 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:41.831201 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:41.831156 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:41.883623 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:41.883593 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:52.768021 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:52.767991 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbdc6748b-txgfj"] Sep 13 20:04:52.868341 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:52.868309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:52.967935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:52.967906 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:04:53.018959 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:04:53.018903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:17.791570 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:17.791527 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fbdc6748b-txgfj" podUID="24788071-b196-4971-bf07-f0166dda2ce8" containerName="console" containerID="cri-o://945a8c838ac8b7fa4cb6306d05529b077aac206cd331495a898a1024388154c5" gracePeriod=15 Sep 13 20:05:17.997765 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:17.997730 2578 generic.go:356] "Generic (PLEG): container finished" podID="24788071-b196-4971-bf07-f0166dda2ce8" containerID="945a8c838ac8b7fa4cb6306d05529b077aac206cd331495a898a1024388154c5" exitCode=0 Sep 13 20:05:17.997921 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:17.997810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdc6748b-txgfj" event={"ID":"24788071-b196-4971-bf07-f0166dda2ce8","Type":"ContainerDied","Data":"945a8c838ac8b7fa4cb6306d05529b077aac206cd331495a898a1024388154c5"} Sep 13 20:05:18.025288 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.025260 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:05:18.126524 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126435 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-console-config\") pod \"24788071-b196-4971-bf07-f0166dda2ce8\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " Sep 13 20:05:18.126524 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdtm5\" (UniqueName: \"kubernetes.io/projected/24788071-b196-4971-bf07-f0166dda2ce8-kube-api-access-pdtm5\") pod \"24788071-b196-4971-bf07-f0166dda2ce8\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " Sep 13 20:05:18.126524 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126516 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-oauth-serving-cert\") pod \"24788071-b196-4971-bf07-f0166dda2ce8\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " Sep 13 20:05:18.126850 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126673 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-oauth-config\") pod \"24788071-b196-4971-bf07-f0166dda2ce8\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " Sep 13 20:05:18.126850 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126737 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-serving-cert\") pod \"24788071-b196-4971-bf07-f0166dda2ce8\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " Sep 13 20:05:18.126850 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126762 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-trusted-ca-bundle\") pod \"24788071-b196-4971-bf07-f0166dda2ce8\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " Sep 13 20:05:18.126850 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126797 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-service-ca\") pod \"24788071-b196-4971-bf07-f0166dda2ce8\" (UID: \"24788071-b196-4971-bf07-f0166dda2ce8\") " Sep 13 20:05:18.126850 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126838 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-console-config" (OuterVolumeSpecName: "console-config") pod "24788071-b196-4971-bf07-f0166dda2ce8" (UID: "24788071-b196-4971-bf07-f0166dda2ce8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 13 20:05:18.127087 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.126908 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "24788071-b196-4971-bf07-f0166dda2ce8" (UID: "24788071-b196-4971-bf07-f0166dda2ce8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 13 20:05:18.127220 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.127174 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-oauth-serving-cert\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:05:18.127291 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.127226 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-console-config\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:05:18.127291 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.127175 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "24788071-b196-4971-bf07-f0166dda2ce8" (UID: "24788071-b196-4971-bf07-f0166dda2ce8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 13 20:05:18.127291 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.127206 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-service-ca" (OuterVolumeSpecName: "service-ca") pod "24788071-b196-4971-bf07-f0166dda2ce8" (UID: "24788071-b196-4971-bf07-f0166dda2ce8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 13 20:05:18.128897 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.128871 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "24788071-b196-4971-bf07-f0166dda2ce8" (UID: "24788071-b196-4971-bf07-f0166dda2ce8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 13 20:05:18.128961 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.128926 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24788071-b196-4971-bf07-f0166dda2ce8-kube-api-access-pdtm5" (OuterVolumeSpecName: "kube-api-access-pdtm5") pod "24788071-b196-4971-bf07-f0166dda2ce8" (UID: "24788071-b196-4971-bf07-f0166dda2ce8"). InnerVolumeSpecName "kube-api-access-pdtm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:05:18.129000 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.128962 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "24788071-b196-4971-bf07-f0166dda2ce8" (UID: "24788071-b196-4971-bf07-f0166dda2ce8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 13 20:05:18.228574 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.228533 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtm5\" (UniqueName: \"kubernetes.io/projected/24788071-b196-4971-bf07-f0166dda2ce8-kube-api-access-pdtm5\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:05:18.228574 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.228566 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-oauth-config\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:05:18.228574 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.228578 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24788071-b196-4971-bf07-f0166dda2ce8-console-serving-cert\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:05:18.228794 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.228589 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-trusted-ca-bundle\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:05:18.228794 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:18.228601 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24788071-b196-4971-bf07-f0166dda2ce8-service-ca\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:05:19.003077 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:19.003043 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbdc6748b-txgfj" Sep 13 20:05:19.003474 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:19.003039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbdc6748b-txgfj" event={"ID":"24788071-b196-4971-bf07-f0166dda2ce8","Type":"ContainerDied","Data":"d132b773d5e629c2e9293139d2f2a1359eb3bf6ad1ae2807d422e8071207ccbe"} Sep 13 20:05:19.003474 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:19.003203 2578 scope.go:117] "RemoveContainer" containerID="945a8c838ac8b7fa4cb6306d05529b077aac206cd331495a898a1024388154c5" Sep 13 20:05:19.018541 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:19.018492 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbdc6748b-txgfj"] Sep 13 20:05:19.020354 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:19.020335 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fbdc6748b-txgfj"] Sep 13 20:05:20.418913 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:20.418874 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24788071-b196-4971-bf07-f0166dda2ce8" path="/var/lib/kubelet/pods/24788071-b196-4971-bf07-f0166dda2ce8/volumes" Sep 13 20:05:31.528348 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:31.528311 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:31.628443 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:31.628411 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:47.624218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:47.624111 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:47.677134 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:47.677104 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:53.346110 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:53.346073 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:53.395492 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:53.395457 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:53.491559 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:53.491530 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:53.592303 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:53.592272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:54.092089 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:54.092056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:54.139098 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:54.139066 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:59.501673 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:59.501642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:05:59.602548 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:05:59.602514 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:04.105016 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:04.104959 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:04.207586 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:04.207558 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:04.256369 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:04.256345 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:12.840351 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:12.840318 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:12.938355 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:12.938325 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:13.723281 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:13.723236 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:27.128624 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:27.128587 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:27.229363 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:27.229329 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:56.203981 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:56.203943 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:06:56.299045 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:06:56.299015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:22.852233 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:22.852146 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:23.611301 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:23.611270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:26.220367 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:26.220338 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:26.274750 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:26.274718 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:28.831502 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:28.831466 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:28.935620 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:28.935587 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:29.060326 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.060295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:29.150951 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.150880 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:29.184342 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.184289 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pfx8z"] Sep 13 20:07:29.184610 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.184598 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24788071-b196-4971-bf07-f0166dda2ce8" containerName="console" Sep 13 20:07:29.184659 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.184612 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24788071-b196-4971-bf07-f0166dda2ce8" containerName="console" Sep 13 20:07:29.184701 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.184658 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24788071-b196-4971-bf07-f0166dda2ce8" containerName="console" Sep 13 20:07:29.187440 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.187424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.189561 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.189535 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 13 20:07:29.190271 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.190252 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 13 20:07:29.190354 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.190256 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mws47\"" Sep 13 20:07:29.197039 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.197013 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pfx8z"] Sep 13 20:07:29.321299 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.321261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgbf\" (UniqueName: \"kubernetes.io/projected/b6e770f6-69f4-4f42-987e-76a70a04a938-kube-api-access-jsgbf\") pod \"cert-manager-webhook-6486ffd997-pfx8z\" (UID: \"b6e770f6-69f4-4f42-987e-76a70a04a938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.321464 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.321323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6e770f6-69f4-4f42-987e-76a70a04a938-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pfx8z\" (UID: \"b6e770f6-69f4-4f42-987e-76a70a04a938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.422246 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.422144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgbf\" (UniqueName: \"kubernetes.io/projected/b6e770f6-69f4-4f42-987e-76a70a04a938-kube-api-access-jsgbf\") pod \"cert-manager-webhook-6486ffd997-pfx8z\" (UID: \"b6e770f6-69f4-4f42-987e-76a70a04a938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.422246 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.422210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6e770f6-69f4-4f42-987e-76a70a04a938-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pfx8z\" (UID: \"b6e770f6-69f4-4f42-987e-76a70a04a938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.434611 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.434578 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgbf\" (UniqueName: \"kubernetes.io/projected/b6e770f6-69f4-4f42-987e-76a70a04a938-kube-api-access-jsgbf\") pod \"cert-manager-webhook-6486ffd997-pfx8z\" (UID: \"b6e770f6-69f4-4f42-987e-76a70a04a938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.434722 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.434611 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6e770f6-69f4-4f42-987e-76a70a04a938-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pfx8z\" (UID: \"b6e770f6-69f4-4f42-987e-76a70a04a938\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.496050 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.496021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:29.766947 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:29.766922 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pfx8z"] Sep 13 20:07:29.769470 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:07:29.769440 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6e770f6_69f4_4f42_987e_76a70a04a938.slice/crio-04ed5b3d39430b77e3628fd19d24eeb6a7053b0004d79ccc599c39bb13456ca3 WatchSource:0}: Error finding container 04ed5b3d39430b77e3628fd19d24eeb6a7053b0004d79ccc599c39bb13456ca3: Status 404 returned error can't find the container with id 04ed5b3d39430b77e3628fd19d24eeb6a7053b0004d79ccc599c39bb13456ca3 Sep 13 20:07:30.483768 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:30.483734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" event={"ID":"b6e770f6-69f4-4f42-987e-76a70a04a938","Type":"ContainerStarted","Data":"04ed5b3d39430b77e3628fd19d24eeb6a7053b0004d79ccc599c39bb13456ca3"} Sep 13 20:07:31.221031 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:31.220975 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:31.653555 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:31.653485 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:33.222142 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:33.222049 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:33.277436 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:33.277400 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:33.498051 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:33.497964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" event={"ID":"b6e770f6-69f4-4f42-987e-76a70a04a938","Type":"ContainerStarted","Data":"2a7b1c7d96056f8f9d0ab56b1da498fd585c7e12a57d39810a3f22896867f589"} Sep 13 20:07:33.498051 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:33.498024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:33.513684 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:33.513635 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" podStartSLOduration=1.330604146 podStartE2EDuration="4.513618097s" totalImagesPullingTime="3.183013951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:07:33.511949629 +0000 UTC m=+549.595094548" watchObservedRunningTime="2026-09-13 20:07:33.513618097 +0000 UTC m=+549.596763010" Sep 13 20:07:39.503909 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:39.503878 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pfx8z" Sep 13 20:07:39.615528 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:39.615499 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:42.366333 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:42.366295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:42.471906 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:42.471871 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:43.627143 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:43.627105 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:43.727395 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:43.727361 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:45.851147 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.851105 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk"] Sep 13 20:07:45.857560 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.857541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:45.860303 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.860276 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 13 20:07:45.860440 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.860321 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k8ckj\"" Sep 13 20:07:45.860617 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.860574 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 13 20:07:45.861083 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.861062 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk"] Sep 13 20:07:45.960868 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.960834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvhv2\" (UniqueName: \"kubernetes.io/projected/307e3d7e-7cd6-49a7-9845-79a1aa12ed22-kube-api-access-lvhv2\") pod \"openshift-lws-operator-fd8ccff4c-kgzrk\" (UID: \"307e3d7e-7cd6-49a7-9845-79a1aa12ed22\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:45.960868 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:45.960871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/307e3d7e-7cd6-49a7-9845-79a1aa12ed22-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kgzrk\" (UID: \"307e3d7e-7cd6-49a7-9845-79a1aa12ed22\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:46.061807 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:46.061761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvhv2\" (UniqueName: \"kubernetes.io/projected/307e3d7e-7cd6-49a7-9845-79a1aa12ed22-kube-api-access-lvhv2\") pod \"openshift-lws-operator-fd8ccff4c-kgzrk\" (UID: \"307e3d7e-7cd6-49a7-9845-79a1aa12ed22\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:46.062027 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:46.061821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/307e3d7e-7cd6-49a7-9845-79a1aa12ed22-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kgzrk\" (UID: \"307e3d7e-7cd6-49a7-9845-79a1aa12ed22\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:46.062250 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:46.062226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/307e3d7e-7cd6-49a7-9845-79a1aa12ed22-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kgzrk\" (UID: \"307e3d7e-7cd6-49a7-9845-79a1aa12ed22\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:46.069841 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:46.069817 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvhv2\" (UniqueName: \"kubernetes.io/projected/307e3d7e-7cd6-49a7-9845-79a1aa12ed22-kube-api-access-lvhv2\") pod \"openshift-lws-operator-fd8ccff4c-kgzrk\" (UID: \"307e3d7e-7cd6-49a7-9845-79a1aa12ed22\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:46.167083 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:46.167022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" Sep 13 20:07:46.438153 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:46.438082 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk"] Sep 13 20:07:46.440894 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:07:46.440869 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod307e3d7e_7cd6_49a7_9845_79a1aa12ed22.slice/crio-9e9f718a8aaae9d4aa29296e1b0a5d3adcb3dfe24edc84bfa8a982df237ef7c6 WatchSource:0}: Error finding container 9e9f718a8aaae9d4aa29296e1b0a5d3adcb3dfe24edc84bfa8a982df237ef7c6: Status 404 returned error can't find the container with id 9e9f718a8aaae9d4aa29296e1b0a5d3adcb3dfe24edc84bfa8a982df237ef7c6 Sep 13 20:07:46.550009 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:46.549981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" event={"ID":"307e3d7e-7cd6-49a7-9845-79a1aa12ed22","Type":"ContainerStarted","Data":"9e9f718a8aaae9d4aa29296e1b0a5d3adcb3dfe24edc84bfa8a982df237ef7c6"} Sep 13 20:07:49.557999 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:49.557970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:49.564785 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:49.564754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" event={"ID":"307e3d7e-7cd6-49a7-9845-79a1aa12ed22","Type":"ContainerStarted","Data":"c98dbd55bcd1a5145d885a0c5a6cb7ca68bb63ccf88ddc8d30d465528c60075c"} Sep 13 20:07:49.582380 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:49.582330 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kgzrk" podStartSLOduration=2.311901672 podStartE2EDuration="4.582316629s" totalImagesPullingTime="2.270414957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:07:49.581322264 +0000 UTC m=+565.664467179" watchObservedRunningTime="2026-09-13 20:07:49.582316629 +0000 UTC m=+565.665461542" Sep 13 20:07:50.270963 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:50.270931 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:50.374633 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:50.374595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:07:52.114717 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:07:52.114679 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:04.529877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:04.529823 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:04.631608 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:04.631558 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:05.155706 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.155675 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:05.348273 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.348233 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:05.445984 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.445904 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:05.485549 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.485520 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf"] Sep 13 20:08:05.490744 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.490726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.493152 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.493125 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 13 20:08:05.493275 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.493178 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 13 20:08:05.493275 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.493252 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 13 20:08:05.493383 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.493256 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 13 20:08:05.493425 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.493386 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lh2xm\"" Sep 13 20:08:05.502997 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.502971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4288a23-4050-49a9-87b8-490cf144ff50-webhook-cert\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.503119 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.503080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4288a23-4050-49a9-87b8-490cf144ff50-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.503220 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.503117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4288a23-4050-49a9-87b8-490cf144ff50-manifests\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.503220 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.503149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4288a23-4050-49a9-87b8-490cf144ff50-tmp\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.503323 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.503215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9k4\" (UniqueName: \"kubernetes.io/projected/b4288a23-4050-49a9-87b8-490cf144ff50-kube-api-access-2n9k4\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.504035 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.504014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf"] Sep 13 20:08:05.604170 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.604133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9k4\" (UniqueName: \"kubernetes.io/projected/b4288a23-4050-49a9-87b8-490cf144ff50-kube-api-access-2n9k4\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.604569 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.604217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4288a23-4050-49a9-87b8-490cf144ff50-webhook-cert\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.604569 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.604296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4288a23-4050-49a9-87b8-490cf144ff50-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.604569 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.604328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4288a23-4050-49a9-87b8-490cf144ff50-manifests\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.604569 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.604359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4288a23-4050-49a9-87b8-490cf144ff50-tmp\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.604804 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.604782 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4288a23-4050-49a9-87b8-490cf144ff50-manifests\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.606521 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.606499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4288a23-4050-49a9-87b8-490cf144ff50-tmp\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.606776 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.606759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4288a23-4050-49a9-87b8-490cf144ff50-webhook-cert\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.606931 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.606911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4288a23-4050-49a9-87b8-490cf144ff50-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.621600 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.621565 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9k4\" (UniqueName: \"kubernetes.io/projected/b4288a23-4050-49a9-87b8-490cf144ff50-kube-api-access-2n9k4\") pod \"opendatahub-operator-controller-manager-79dc4cfddb-phnhf\" (UID: \"b4288a23-4050-49a9-87b8-490cf144ff50\") " pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.639490 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.639461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:05.743860 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.743783 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:05.802217 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.802171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:05.840075 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.840037 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:05.943029 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:05.942997 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:06.039452 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:06.039424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:06.085280 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:06.085109 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf"] Sep 13 20:08:06.087727 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:08:06.087699 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4288a23_4050_49a9_87b8_490cf144ff50.slice/crio-4c6059cff011e488c8ff4a22697627ef8f753e95f8b1922c0916dc46139e0ab0 WatchSource:0}: Error finding container 4c6059cff011e488c8ff4a22697627ef8f753e95f8b1922c0916dc46139e0ab0: Status 404 returned error can't find the container with id 4c6059cff011e488c8ff4a22697627ef8f753e95f8b1922c0916dc46139e0ab0 Sep 13 20:08:06.142056 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:06.142019 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:06.240166 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:06.240136 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:06.633954 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:06.633918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" event={"ID":"b4288a23-4050-49a9-87b8-490cf144ff50","Type":"ContainerStarted","Data":"4c6059cff011e488c8ff4a22697627ef8f753e95f8b1922c0916dc46139e0ab0"} Sep 13 20:08:09.656027 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:09.655974 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" event={"ID":"b4288a23-4050-49a9-87b8-490cf144ff50","Type":"ContainerStarted","Data":"5beaa35439ca53be30eb33a29ce6cb8e63e6f2a55f3d74ac3811f73ebc8afb01"} Sep 13 20:08:10.384717 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:10.384674 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:10.438150 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:10.438059 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:10.662259 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:10.662225 2578 generic.go:356] "Generic (PLEG): container finished" podID="b4288a23-4050-49a9-87b8-490cf144ff50" containerID="5beaa35439ca53be30eb33a29ce6cb8e63e6f2a55f3d74ac3811f73ebc8afb01" exitCode=0 Sep 13 20:08:10.662647 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:10.662282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" event={"ID":"b4288a23-4050-49a9-87b8-490cf144ff50","Type":"ContainerDied","Data":"5beaa35439ca53be30eb33a29ce6cb8e63e6f2a55f3d74ac3811f73ebc8afb01"} Sep 13 20:08:11.040202 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:11.040162 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:11.601391 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:11.601353 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:11.648750 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:11.648721 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:11.668652 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:11.668627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" event={"ID":"b4288a23-4050-49a9-87b8-490cf144ff50","Type":"ContainerStarted","Data":"2e9131f6e93232939547bcc24634e3df4254fa1edcb39a315106133d7f9739e1"} Sep 13 20:08:11.669063 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:11.668735 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:11.688723 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:11.688671 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" podStartSLOduration=2.289963345 podStartE2EDuration="6.688655518s" totalImagesPullingTime="3.304309288s" totalInitContainerRuntime="1.094382885s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:08:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:08:11.686555136 +0000 UTC m=+587.769700051" watchObservedRunningTime="2026-09-13 20:08:11.688655518 +0000 UTC m=+587.771800432" Sep 13 20:08:16.211813 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:16.211779 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:16.320008 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:16.319979 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:17.281970 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:17.281937 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:17.383124 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:17.383095 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:22.675180 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:22.675148 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79dc4cfddb-phnhf" Sep 13 20:08:24.218887 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.218849 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:24.462488 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.462390 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d4d676b-vvxlj"] Sep 13 20:08:24.528653 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.528594 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d4d676b-vvxlj"] Sep 13 20:08:24.528748 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.528677 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.530964 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.530938 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 13 20:08:24.531102 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.531041 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n7dxr\"" Sep 13 20:08:24.531102 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.531064 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 13 20:08:24.531239 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.531156 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 13 20:08:24.531730 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.531694 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 13 20:08:24.662975 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.662931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tmp\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.662975 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.662972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tls-certs\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.663237 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.663004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljf5z\" (UniqueName: \"kubernetes.io/projected/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-kube-api-access-ljf5z\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.763823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.763786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tmp\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.763823 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.763824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tls-certs\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.764005 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.763860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljf5z\" (UniqueName: \"kubernetes.io/projected/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-kube-api-access-ljf5z\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.764005 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:08:24.763962 2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 13 20:08:24.764077 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:08:24.764067 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tls-certs podName:3fe165cf-58f8-4dea-82e3-d6e1d48ae04a nodeName:}" failed. No retries permitted until 2026-09-13 20:08:25.264044851 +0000 UTC m=+601.347189762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tls-certs") pod "kube-auth-proxy-d4d676b-vvxlj" (UID: "3fe165cf-58f8-4dea-82e3-d6e1d48ae04a") : secret "kube-auth-proxy-tls" not found Sep 13 20:08:24.765920 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.765899 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tmp\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:24.772255 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:24.772229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljf5z\" (UniqueName: \"kubernetes.io/projected/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-kube-api-access-ljf5z\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:25.268564 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.268529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tls-certs\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:25.270935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.270913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe165cf-58f8-4dea-82e3-d6e1d48ae04a-tls-certs\") pod \"kube-auth-proxy-d4d676b-vvxlj\" (UID: \"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a\") " pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:25.438265 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.438227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" Sep 13 20:08:25.654107 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.654014 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:25.657861 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.657837 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb"] Sep 13 20:08:25.660776 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.660757 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.662904 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.662869 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 13 20:08:25.662904 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.662886 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fxmf6\"" Sep 13 20:08:25.663052 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.662909 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 13 20:08:25.663052 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.662869 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 13 20:08:25.663052 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.662886 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 13 20:08:25.674124 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.674101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb"] Sep 13 20:08:25.710702 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.710681 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d4d676b-vvxlj"] Sep 13 20:08:25.713019 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:08:25.712986 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe165cf_58f8_4dea_82e3_d6e1d48ae04a.slice/crio-79fa527fa3002c7c4300b3460e26b6d4173ea5f6cd60cae73bdcc9bafbb3c24d WatchSource:0}: Error finding container 79fa527fa3002c7c4300b3460e26b6d4173ea5f6cd60cae73bdcc9bafbb3c24d: Status 404 returned error can't find the container with id 79fa527fa3002c7c4300b3460e26b6d4173ea5f6cd60cae73bdcc9bafbb3c24d Sep 13 20:08:25.716135 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.716097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:25.732833 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.732804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" event={"ID":"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a","Type":"ContainerStarted","Data":"79fa527fa3002c7c4300b3460e26b6d4173ea5f6cd60cae73bdcc9bafbb3c24d"} Sep 13 20:08:25.772220 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.772197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.772296 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.772269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.772341 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.772292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.772341 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.772315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.772452 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.772339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jzx\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-kube-api-access-v2jzx\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.772452 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.772375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.772452 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.772405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.873411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.873387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.873494 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.873415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.873494 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.873446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.873494 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.873471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.873494 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.873488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.873667 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.873502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.873667 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.873521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jzx\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-kube-api-access-v2jzx\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.874123 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.874102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.875658 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.875636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.875727 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.875656 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.875978 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.875961 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.876047 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.876022 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.880670 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.880645 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jzx\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-kube-api-access-v2jzx\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.880670 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.880655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-42mhb\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:25.969763 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:25.969713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:26.239427 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:26.239396 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb"] Sep 13 20:08:26.241364 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:08:26.241336 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f19f58e_aa6f_47ba_9be6_3e0b12238f24.slice/crio-882a9f67e76114a19b2fb9a73849e14dbb6da440cbb8f50d2ed10952edffde01 WatchSource:0}: Error finding container 882a9f67e76114a19b2fb9a73849e14dbb6da440cbb8f50d2ed10952edffde01: Status 404 returned error can't find the container with id 882a9f67e76114a19b2fb9a73849e14dbb6da440cbb8f50d2ed10952edffde01 Sep 13 20:08:26.484219 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:26.484167 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:26.740152 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:26.740107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" event={"ID":"9f19f58e-aa6f-47ba-9be6-3e0b12238f24","Type":"ContainerStarted","Data":"882a9f67e76114a19b2fb9a73849e14dbb6da440cbb8f50d2ed10952edffde01"} Sep 13 20:08:27.737648 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:27.737604 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:27.786816 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:27.786777 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:27.841386 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:27.841352 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:29.802945 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:29.802912 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 13 20:08:29.803338 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:29.802976 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 13 20:08:30.763906 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.763861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" event={"ID":"9f19f58e-aa6f-47ba-9be6-3e0b12238f24","Type":"ContainerStarted","Data":"5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450"} Sep 13 20:08:30.764218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.764039 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:30.766009 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.765983 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-42mhb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 13 20:08:30.766297 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.766272 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" podUID="9f19f58e-aa6f-47ba-9be6-3e0b12238f24" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 13 20:08:30.766448 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.766109 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" event={"ID":"3fe165cf-58f8-4dea-82e3-d6e1d48ae04a","Type":"ContainerStarted","Data":"6afde25e2f26cdef98a256b9fa19a29a785ba3c197ee96719faa04fd52243c20"} Sep 13 20:08:30.789175 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.789102 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" podStartSLOduration=2.229805497 podStartE2EDuration="5.789086675s" totalImagesPullingTime="3.559281178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:08:30.786350432 +0000 UTC m=+606.869495375" watchObservedRunningTime="2026-09-13 20:08:30.789086675 +0000 UTC m=+606.872231589" Sep 13 20:08:30.797221 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.797172 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:30.802233 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.802167 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d4d676b-vvxlj" podStartSLOduration=2.764167562 podStartE2EDuration="6.802154362s" totalImagesPullingTime="4.0379868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:08:30.801242866 +0000 UTC m=+606.884387806" watchObservedRunningTime="2026-09-13 20:08:30.802154362 +0000 UTC m=+606.885299278" Sep 13 20:08:30.853989 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:30.853958 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:31.376269 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:31.376230 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:31.569352 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:31.569322 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:31.702642 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:31.702566 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:31.770546 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:31.770520 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:35.260270 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:35.260237 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:35.317068 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:35.317037 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:35.417707 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:35.417675 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:35.517866 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:35.517789 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:35.793353 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:35.793275 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:35.948133 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:35.948095 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:44.077177 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.077139 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:44.123113 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.123074 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-97lfk"] Sep 13 20:08:44.127054 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.127029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.129032 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.129008 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs"] Sep 13 20:08:44.129716 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.129489 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dmvp4\"" Sep 13 20:08:44.129716 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.129622 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 13 20:08:44.132348 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.132330 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.134425 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.134401 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 13 20:08:44.134634 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.134615 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-szr9r\"" Sep 13 20:08:44.137645 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.137621 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-97lfk"] Sep 13 20:08:44.141131 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.141106 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs"] Sep 13 20:08:44.154811 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.154781 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:44.231709 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.231669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7571ab8-b66f-4aa5-b026-def4f907dd22-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rb8rs\" (UID: \"a7571ab8-b66f-4aa5-b026-def4f907dd22\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.231709 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.231711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52c98\" (UniqueName: \"kubernetes.io/projected/2ea06763-470f-4bb3-b103-0165a34d72ba-kube-api-access-52c98\") pod \"kserve-controller-manager-d6f4b65b8-97lfk\" (UID: \"2ea06763-470f-4bb3-b103-0165a34d72ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.231920 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.231751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgk6m\" (UniqueName: \"kubernetes.io/projected/a7571ab8-b66f-4aa5-b026-def4f907dd22-kube-api-access-dgk6m\") pod \"llmisvc-controller-manager-777c6bd7d5-rb8rs\" (UID: \"a7571ab8-b66f-4aa5-b026-def4f907dd22\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.231920 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.231809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ea06763-470f-4bb3-b103-0165a34d72ba-cert\") pod \"kserve-controller-manager-d6f4b65b8-97lfk\" (UID: \"2ea06763-470f-4bb3-b103-0165a34d72ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.244074 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.244047 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:44.332857 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.332779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ea06763-470f-4bb3-b103-0165a34d72ba-cert\") pod \"kserve-controller-manager-d6f4b65b8-97lfk\" (UID: \"2ea06763-470f-4bb3-b103-0165a34d72ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.332857 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.332834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7571ab8-b66f-4aa5-b026-def4f907dd22-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rb8rs\" (UID: \"a7571ab8-b66f-4aa5-b026-def4f907dd22\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.333021 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.332864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52c98\" (UniqueName: \"kubernetes.io/projected/2ea06763-470f-4bb3-b103-0165a34d72ba-kube-api-access-52c98\") pod \"kserve-controller-manager-d6f4b65b8-97lfk\" (UID: \"2ea06763-470f-4bb3-b103-0165a34d72ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.333021 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.332892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgk6m\" (UniqueName: \"kubernetes.io/projected/a7571ab8-b66f-4aa5-b026-def4f907dd22-kube-api-access-dgk6m\") pod \"llmisvc-controller-manager-777c6bd7d5-rb8rs\" (UID: \"a7571ab8-b66f-4aa5-b026-def4f907dd22\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.335429 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.335402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ea06763-470f-4bb3-b103-0165a34d72ba-cert\") pod \"kserve-controller-manager-d6f4b65b8-97lfk\" (UID: \"2ea06763-470f-4bb3-b103-0165a34d72ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.335633 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.335613 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7571ab8-b66f-4aa5-b026-def4f907dd22-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rb8rs\" (UID: \"a7571ab8-b66f-4aa5-b026-def4f907dd22\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.343154 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.343129 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52c98\" (UniqueName: \"kubernetes.io/projected/2ea06763-470f-4bb3-b103-0165a34d72ba-kube-api-access-52c98\") pod \"kserve-controller-manager-d6f4b65b8-97lfk\" (UID: \"2ea06763-470f-4bb3-b103-0165a34d72ba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.343462 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.343445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgk6m\" (UniqueName: \"kubernetes.io/projected/a7571ab8-b66f-4aa5-b026-def4f907dd22-kube-api-access-dgk6m\") pod \"llmisvc-controller-manager-777c6bd7d5-rb8rs\" (UID: \"a7571ab8-b66f-4aa5-b026-def4f907dd22\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.438105 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.438072 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:44.443082 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.443067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:44.450310 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.450276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:44.730336 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.730311 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-97lfk"] Sep 13 20:08:44.732100 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:08:44.732071 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea06763_470f_4bb3_b103_0165a34d72ba.slice/crio-90bf51623ac2bdc5a8b115ed4f6392793d652a78e621bf8efb812cf67e0b94b9 WatchSource:0}: Error finding container 90bf51623ac2bdc5a8b115ed4f6392793d652a78e621bf8efb812cf67e0b94b9: Status 404 returned error can't find the container with id 90bf51623ac2bdc5a8b115ed4f6392793d652a78e621bf8efb812cf67e0b94b9 Sep 13 20:08:44.733572 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.733554 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 13 20:08:44.739553 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.739533 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs"] Sep 13 20:08:44.741349 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:08:44.741328 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda7571ab8_b66f_4aa5_b026_def4f907dd22.slice/crio-cf94e1e96a746193a04236bebaca398d63258df30bbaedf72ef34cd7a0538c1f WatchSource:0}: Error finding container cf94e1e96a746193a04236bebaca398d63258df30bbaedf72ef34cd7a0538c1f: Status 404 returned error can't find the container with id cf94e1e96a746193a04236bebaca398d63258df30bbaedf72ef34cd7a0538c1f Sep 13 20:08:44.825879 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.825845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" event={"ID":"a7571ab8-b66f-4aa5-b026-def4f907dd22","Type":"ContainerStarted","Data":"cf94e1e96a746193a04236bebaca398d63258df30bbaedf72ef34cd7a0538c1f"} Sep 13 20:08:44.826935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:44.826912 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" event={"ID":"2ea06763-470f-4bb3-b103-0165a34d72ba","Type":"ContainerStarted","Data":"90bf51623ac2bdc5a8b115ed4f6392793d652a78e621bf8efb812cf67e0b94b9"} Sep 13 20:08:45.329495 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:45.329465 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:45.440512 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:45.440451 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:45.551802 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:45.551722 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:08:45.922439 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:45.922404 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb"] Sep 13 20:08:45.922683 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:45.922651 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" podUID="9f19f58e-aa6f-47ba-9be6-3e0b12238f24" containerName="discovery" containerID="cri-o://5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450" gracePeriod=30 Sep 13 20:08:46.258941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.258914 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:46.350339 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350304 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-ca-configmap\") pod \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350363 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-kubeconfig\") pod \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350388 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-token\") pod \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350403 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-cacerts\") pod \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-local-certs\") pod \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350488 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-dns-cert\") pod \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350561 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2jzx\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-kube-api-access-v2jzx\") pod \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\" (UID: \"9f19f58e-aa6f-47ba-9be6-3e0b12238f24\") " Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350715 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9f19f58e-aa6f-47ba-9be6-3e0b12238f24" (UID: "9f19f58e-aa6f-47ba-9be6-3e0b12238f24"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 13 20:08:46.350822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.350825 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-ca-configmap\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:08:46.353779 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.353439 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-local-certs" (OuterVolumeSpecName: "local-certs") pod "9f19f58e-aa6f-47ba-9be6-3e0b12238f24" (UID: "9f19f58e-aa6f-47ba-9be6-3e0b12238f24"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 13 20:08:46.353779 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.353443 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9f19f58e-aa6f-47ba-9be6-3e0b12238f24" (UID: "9f19f58e-aa6f-47ba-9be6-3e0b12238f24"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 13 20:08:46.353779 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.353578 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-kube-api-access-v2jzx" (OuterVolumeSpecName: "kube-api-access-v2jzx") pod "9f19f58e-aa6f-47ba-9be6-3e0b12238f24" (UID: "9f19f58e-aa6f-47ba-9be6-3e0b12238f24"). InnerVolumeSpecName "kube-api-access-v2jzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:08:46.353779 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.353694 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f19f58e-aa6f-47ba-9be6-3e0b12238f24" (UID: "9f19f58e-aa6f-47ba-9be6-3e0b12238f24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:08:46.354088 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.353901 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9f19f58e-aa6f-47ba-9be6-3e0b12238f24" (UID: "9f19f58e-aa6f-47ba-9be6-3e0b12238f24"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 13 20:08:46.354580 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.354559 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-cacerts" (OuterVolumeSpecName: "cacerts") pod "9f19f58e-aa6f-47ba-9be6-3e0b12238f24" (UID: "9f19f58e-aa6f-47ba-9be6-3e0b12238f24"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 13 20:08:46.451984 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.451953 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jzx\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-kube-api-access-v2jzx\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:08:46.451984 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.451979 2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-kubeconfig\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:08:46.451984 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.451989 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-token\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:08:46.452218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.451997 2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-cacerts\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:08:46.452218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.452005 2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-local-certs\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:08:46.452218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.452013 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f19f58e-aa6f-47ba-9be6-3e0b12238f24-istio-csr-dns-cert\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:08:46.840676 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.840524 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f19f58e-aa6f-47ba-9be6-3e0b12238f24" containerID="5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450" exitCode=0 Sep 13 20:08:46.840676 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.840602 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" Sep 13 20:08:46.840676 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.840632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" event={"ID":"9f19f58e-aa6f-47ba-9be6-3e0b12238f24","Type":"ContainerDied","Data":"5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450"} Sep 13 20:08:46.840953 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.840696 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb" event={"ID":"9f19f58e-aa6f-47ba-9be6-3e0b12238f24","Type":"ContainerDied","Data":"882a9f67e76114a19b2fb9a73849e14dbb6da440cbb8f50d2ed10952edffde01"} Sep 13 20:08:46.840953 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.840725 2578 scope.go:117] "RemoveContainer" containerID="5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450" Sep 13 20:08:46.857881 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.857851 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb"] Sep 13 20:08:46.864637 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:46.864610 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-42mhb"] Sep 13 20:08:47.205830 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:47.205670 2578 scope.go:117] "RemoveContainer" containerID="5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450" Sep 13 20:08:47.206104 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:08:47.206073 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450\": container with ID starting with 5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450 not found: ID does not exist" containerID="5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450" Sep 13 20:08:47.206151 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:47.206116 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450"} err="failed to get container status \"5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450\": rpc error: code = NotFound desc = could not find container \"5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450\": container with ID starting with 5ec589e6473ba01e98dfb1ee7e438f97eebf641bd38ed0cacbf23266f1054450 not found: ID does not exist" Sep 13 20:08:48.419734 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:48.419699 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f19f58e-aa6f-47ba-9be6-3e0b12238f24" path="/var/lib/kubelet/pods/9f19f58e-aa6f-47ba-9be6-3e0b12238f24/volumes" Sep 13 20:08:48.857444 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:48.857405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" event={"ID":"2ea06763-470f-4bb3-b103-0165a34d72ba","Type":"ContainerStarted","Data":"72843b6829ab44dede0cd7ca52cbc44e306dfa0f8a124a0c4691cce9b9d39aed"} Sep 13 20:08:48.857657 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:48.857558 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:08:48.858877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:48.858856 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" event={"ID":"a7571ab8-b66f-4aa5-b026-def4f907dd22","Type":"ContainerStarted","Data":"c5717003843221e5e4920d296d767a3a86dbce0cf6b6f6097eb86b03ca8e3353"} Sep 13 20:08:48.858989 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:48.858976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:08:48.876156 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:48.876110 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" podStartSLOduration=1.826603799 podStartE2EDuration="4.876097029s" totalImagesPullingTime="3.04949323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:08:48.875250132 +0000 UTC m=+624.958395050" watchObservedRunningTime="2026-09-13 20:08:48.876097029 +0000 UTC m=+624.959241944" Sep 13 20:08:48.892596 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:08:48.892540 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" podStartSLOduration=1.848116454 podStartE2EDuration="4.892522319s" totalImagesPullingTime="3.044405865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:08:48.891112283 +0000 UTC m=+624.974257195" watchObservedRunningTime="2026-09-13 20:08:48.892522319 +0000 UTC m=+624.975667277" Sep 13 20:09:19.865235 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:19.865199 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rb8rs" Sep 13 20:09:19.868399 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:19.868374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-97lfk" Sep 13 20:09:36.306244 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:36.306159 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:36.402895 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:36.402865 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:49.209047 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:49.209015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:51.233438 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.233405 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:51.266624 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.266598 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw"] Sep 13 20:09:51.266920 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.266908 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f19f58e-aa6f-47ba-9be6-3e0b12238f24" containerName="discovery" Sep 13 20:09:51.266962 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.266921 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f19f58e-aa6f-47ba-9be6-3e0b12238f24" containerName="discovery" Sep 13 20:09:51.266995 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.266985 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f19f58e-aa6f-47ba-9be6-3e0b12238f24" containerName="discovery" Sep 13 20:09:51.276166 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.276142 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:09:51.278342 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.278314 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 13 20:09:51.278481 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.278327 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 13 20:09:51.279211 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.279159 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j6mch\"" Sep 13 20:09:51.281111 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.281088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw"] Sep 13 20:09:51.373341 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.373315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgfv\" (UniqueName: \"kubernetes.io/projected/3810297a-f2bd-40c8-a684-4cd63aba83d8-kube-api-access-5kgfv\") pod \"limitador-operator-controller-manager-74495c69f8-ttsqw\" (UID: \"3810297a-f2bd-40c8-a684-4cd63aba83d8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:09:51.474426 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.474399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgfv\" (UniqueName: \"kubernetes.io/projected/3810297a-f2bd-40c8-a684-4cd63aba83d8-kube-api-access-5kgfv\") pod \"limitador-operator-controller-manager-74495c69f8-ttsqw\" (UID: \"3810297a-f2bd-40c8-a684-4cd63aba83d8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:09:51.484100 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.484049 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgfv\" (UniqueName: \"kubernetes.io/projected/3810297a-f2bd-40c8-a684-4cd63aba83d8-kube-api-access-5kgfv\") pod \"limitador-operator-controller-manager-74495c69f8-ttsqw\" (UID: \"3810297a-f2bd-40c8-a684-4cd63aba83d8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:09:51.587314 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.587283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:09:51.859086 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:51.859058 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw"] Sep 13 20:09:51.861779 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:09:51.861752 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3810297a_f2bd_40c8_a684_4cd63aba83d8.slice/crio-6cda6f013c5430a8b503d69e54c301e193dcaeac1cd34cce59ee5b91ded66feb WatchSource:0}: Error finding container 6cda6f013c5430a8b503d69e54c301e193dcaeac1cd34cce59ee5b91ded66feb: Status 404 returned error can't find the container with id 6cda6f013c5430a8b503d69e54c301e193dcaeac1cd34cce59ee5b91ded66feb Sep 13 20:09:52.049671 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:52.049622 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:52.139296 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:52.139209 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" event={"ID":"3810297a-f2bd-40c8-a684-4cd63aba83d8","Type":"ContainerStarted","Data":"6cda6f013c5430a8b503d69e54c301e193dcaeac1cd34cce59ee5b91ded66feb"} Sep 13 20:09:53.269009 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.268972 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn"] Sep 13 20:09:53.272897 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.272874 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" Sep 13 20:09:53.275620 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.275596 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 13 20:09:53.275731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.275697 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mfzkn\"" Sep 13 20:09:53.282475 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.282448 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn"] Sep 13 20:09:53.290114 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.290086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djv6x\" (UniqueName: \"kubernetes.io/projected/b8a0df6f-6057-452f-aa5a-d0453ca6621b-kube-api-access-djv6x\") pod \"dns-operator-controller-manager-7b84cd7c9c-jtlmn\" (UID: \"b8a0df6f-6057-452f-aa5a-d0453ca6621b\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" Sep 13 20:09:53.390742 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.390698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djv6x\" (UniqueName: \"kubernetes.io/projected/b8a0df6f-6057-452f-aa5a-d0453ca6621b-kube-api-access-djv6x\") pod \"dns-operator-controller-manager-7b84cd7c9c-jtlmn\" (UID: \"b8a0df6f-6057-452f-aa5a-d0453ca6621b\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" Sep 13 20:09:53.403225 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.403178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djv6x\" (UniqueName: \"kubernetes.io/projected/b8a0df6f-6057-452f-aa5a-d0453ca6621b-kube-api-access-djv6x\") pod \"dns-operator-controller-manager-7b84cd7c9c-jtlmn\" (UID: \"b8a0df6f-6057-452f-aa5a-d0453ca6621b\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" Sep 13 20:09:53.584505 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.584427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" Sep 13 20:09:53.610403 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:53.610369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:54.044512 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:54.044479 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:54.264080 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:54.264046 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:54.539618 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:54.539527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn"] Sep 13 20:09:54.541686 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:09:54.541662 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a0df6f_6057_452f_aa5a_d0453ca6621b.slice/crio-d06f1f561955b60cc737a8393aae4b34559563f0f4e9b65162a2f45d1bb0094a WatchSource:0}: Error finding container d06f1f561955b60cc737a8393aae4b34559563f0f4e9b65162a2f45d1bb0094a: Status 404 returned error can't find the container with id d06f1f561955b60cc737a8393aae4b34559563f0f4e9b65162a2f45d1bb0094a Sep 13 20:09:55.155655 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:55.155618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" event={"ID":"3810297a-f2bd-40c8-a684-4cd63aba83d8","Type":"ContainerStarted","Data":"7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337"} Sep 13 20:09:55.155941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:55.155777 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:09:55.157365 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:55.157336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" event={"ID":"b8a0df6f-6057-452f-aa5a-d0453ca6621b","Type":"ContainerStarted","Data":"d06f1f561955b60cc737a8393aae4b34559563f0f4e9b65162a2f45d1bb0094a"} Sep 13 20:09:55.177857 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:55.177805 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" podStartSLOduration=1.7032320140000001 podStartE2EDuration="4.177785034s" totalImagesPullingTime="2.47455302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:09:55.174675253 +0000 UTC m=+691.257820167" watchObservedRunningTime="2026-09-13 20:09:55.177785034 +0000 UTC m=+691.260929967" Sep 13 20:09:55.641158 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:55.641074 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:56.078059 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.078015 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cgd8h"] Sep 13 20:09:56.081427 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.081398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" Sep 13 20:09:56.083708 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.083687 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-89nkk\"" Sep 13 20:09:56.095204 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.093485 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cgd8h"] Sep 13 20:09:56.114022 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.113988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4d5d\" (UniqueName: \"kubernetes.io/projected/d4a93942-6044-4477-a5ce-86ad2c74562c-kube-api-access-b4d5d\") pod \"authorino-operator-76b478bd78-cgd8h\" (UID: \"d4a93942-6044-4477-a5ce-86ad2c74562c\") " pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" Sep 13 20:09:56.166817 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.166771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" event={"ID":"b8a0df6f-6057-452f-aa5a-d0453ca6621b","Type":"ContainerStarted","Data":"9fd1cb8afd11c879fba65115f3ff50ec3b1c8bcde665e91f174825826babe363"} Sep 13 20:09:56.167084 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.167058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" Sep 13 20:09:56.193057 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.193005 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" podStartSLOduration=1.7838613570000001 podStartE2EDuration="3.19298608s" totalImagesPullingTime="1.409124723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:09:56.192599309 +0000 UTC m=+692.275744224" watchObservedRunningTime="2026-09-13 20:09:56.19298608 +0000 UTC m=+692.276130994" Sep 13 20:09:56.215248 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.215133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4d5d\" (UniqueName: \"kubernetes.io/projected/d4a93942-6044-4477-a5ce-86ad2c74562c-kube-api-access-b4d5d\") pod \"authorino-operator-76b478bd78-cgd8h\" (UID: \"d4a93942-6044-4477-a5ce-86ad2c74562c\") " pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" Sep 13 20:09:56.224229 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.224176 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4d5d\" (UniqueName: \"kubernetes.io/projected/d4a93942-6044-4477-a5ce-86ad2c74562c-kube-api-access-b4d5d\") pod \"authorino-operator-76b478bd78-cgd8h\" (UID: \"d4a93942-6044-4477-a5ce-86ad2c74562c\") " pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" Sep 13 20:09:56.391853 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.391818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" Sep 13 20:09:56.610765 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.610735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:56.660495 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:56.660468 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cgd8h"] Sep 13 20:09:56.662530 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:09:56.662506 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a93942_6044_4477_a5ce_86ad2c74562c.slice/crio-b56ec16c35d5fde57a387c88d775aa77f7b10785e0b80a9d9cd74dc130e68c21 WatchSource:0}: Error finding container b56ec16c35d5fde57a387c88d775aa77f7b10785e0b80a9d9cd74dc130e68c21: Status 404 returned error can't find the container with id b56ec16c35d5fde57a387c88d775aa77f7b10785e0b80a9d9cd74dc130e68c21 Sep 13 20:09:57.003037 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:57.002955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:57.173936 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:57.173903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" event={"ID":"d4a93942-6044-4477-a5ce-86ad2c74562c","Type":"ContainerStarted","Data":"b56ec16c35d5fde57a387c88d775aa77f7b10785e0b80a9d9cd74dc130e68c21"} Sep 13 20:09:57.432555 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:57.432522 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:57.701531 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:57.701440 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:09:58.180854 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:58.180818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" event={"ID":"d4a93942-6044-4477-a5ce-86ad2c74562c","Type":"ContainerStarted","Data":"64faab7644ba8673c6c318b345a9f60b7d9bd7206bede450743b53db1791f533"} Sep 13 20:09:58.181047 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:58.180945 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" Sep 13 20:09:58.200200 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:58.200132 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" podStartSLOduration=0.832582854 podStartE2EDuration="2.200113103s" totalImagesPullingTime="1.367530249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:09:58.198177418 +0000 UTC m=+694.281322331" watchObservedRunningTime="2026-09-13 20:09:58.200113103 +0000 UTC m=+694.283258030" Sep 13 20:09:59.272627 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:09:59.272595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:00.613328 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.613293 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz"] Sep 13 20:10:00.616901 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.616876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:00.620151 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.620127 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h6fmd\"" Sep 13 20:10:00.628351 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.628327 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz"] Sep 13 20:10:00.655535 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.655494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f2f046-7380-4254-81d2-1e0938e90583-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:00.655696 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.655541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zbqc\" (UniqueName: \"kubernetes.io/projected/e3f2f046-7380-4254-81d2-1e0938e90583-kube-api-access-9zbqc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:00.756963 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.756928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f2f046-7380-4254-81d2-1e0938e90583-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:00.757123 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.756969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zbqc\" (UniqueName: \"kubernetes.io/projected/e3f2f046-7380-4254-81d2-1e0938e90583-kube-api-access-9zbqc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:00.757359 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.757337 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f2f046-7380-4254-81d2-1e0938e90583-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:00.764791 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.764771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zbqc\" (UniqueName: \"kubernetes.io/projected/e3f2f046-7380-4254-81d2-1e0938e90583-kube-api-access-9zbqc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:00.927202 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:00.927092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:01.202108 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:01.202080 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz"] Sep 13 20:10:01.204485 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:10:01.204454 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f2f046_7380_4254_81d2_1e0938e90583.slice/crio-139856d6c1af5f14741c81ff093f1c6e9a84dcceef8ac7686163057072f58644 WatchSource:0}: Error finding container 139856d6c1af5f14741c81ff093f1c6e9a84dcceef8ac7686163057072f58644: Status 404 returned error can't find the container with id 139856d6c1af5f14741c81ff093f1c6e9a84dcceef8ac7686163057072f58644 Sep 13 20:10:01.822662 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:01.822626 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:01.923878 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:01.923821 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:02.203576 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:02.203485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" event={"ID":"e3f2f046-7380-4254-81d2-1e0938e90583","Type":"ContainerStarted","Data":"139856d6c1af5f14741c81ff093f1c6e9a84dcceef8ac7686163057072f58644"} Sep 13 20:10:04.809788 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:04.809745 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:04.872447 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:04.872413 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:06.169604 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:06.169571 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:06.226963 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:06.226933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" event={"ID":"e3f2f046-7380-4254-81d2-1e0938e90583","Type":"ContainerStarted","Data":"05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b"} Sep 13 20:10:06.227135 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:06.227041 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:06.244241 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:06.244169 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" podStartSLOduration=2.253949386 podStartE2EDuration="6.244149674s" totalImagesPullingTime="3.990200288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:10:06.241563914 +0000 UTC m=+702.324708839" watchObservedRunningTime="2026-09-13 20:10:06.244149674 +0000 UTC m=+702.327294589" Sep 13 20:10:07.176089 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.176059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jtlmn" Sep 13 20:10:07.871524 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.871490 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9"] Sep 13 20:10:07.875045 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.875023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:07.877155 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.877128 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 13 20:10:07.877956 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.877940 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4bvtw\"" Sep 13 20:10:07.878056 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.878005 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 13 20:10:07.881371 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.881348 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:07.885931 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.885908 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9"] Sep 13 20:10:07.920662 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.920626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/14afd400-a7ad-4787-8f96-1bde0831d319-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:07.920822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.920754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14afd400-a7ad-4787-8f96-1bde0831d319-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:07.920822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.920793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdg7\" (UniqueName: \"kubernetes.io/projected/14afd400-a7ad-4787-8f96-1bde0831d319-kube-api-access-thdg7\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:07.945359 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:07.945325 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:08.021434 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.021398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14afd400-a7ad-4787-8f96-1bde0831d319-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:08.021629 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.021440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdg7\" (UniqueName: \"kubernetes.io/projected/14afd400-a7ad-4787-8f96-1bde0831d319-kube-api-access-thdg7\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:08.021629 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.021557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/14afd400-a7ad-4787-8f96-1bde0831d319-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:08.022163 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.022135 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14afd400-a7ad-4787-8f96-1bde0831d319-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:08.024041 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.024019 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/14afd400-a7ad-4787-8f96-1bde0831d319-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:08.028411 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.028387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdg7\" (UniqueName: \"kubernetes.io/projected/14afd400-a7ad-4787-8f96-1bde0831d319-kube-api-access-thdg7\") pod \"kuadrant-console-plugin-fbdd4f445-9h2f9\" (UID: \"14afd400-a7ad-4787-8f96-1bde0831d319\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:08.184729 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.184640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" Sep 13 20:10:08.451719 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:08.451694 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9"] Sep 13 20:10:08.453854 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:10:08.453831 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14afd400_a7ad_4787_8f96_1bde0831d319.slice/crio-5e04fbdedc71e8b3efb4517aac9ab9d8a98bfbbc06c091083972c50ce8bd987f WatchSource:0}: Error finding container 5e04fbdedc71e8b3efb4517aac9ab9d8a98bfbbc06c091083972c50ce8bd987f: Status 404 returned error can't find the container with id 5e04fbdedc71e8b3efb4517aac9ab9d8a98bfbbc06c091083972c50ce8bd987f Sep 13 20:10:09.188603 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:09.188572 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-cgd8h" Sep 13 20:10:09.255674 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:09.255633 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" event={"ID":"14afd400-a7ad-4787-8f96-1bde0831d319","Type":"ContainerStarted","Data":"5e04fbdedc71e8b3efb4517aac9ab9d8a98bfbbc06c091083972c50ce8bd987f"} Sep 13 20:10:10.421711 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:10.421682 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:10.521949 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:10.521908 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:13.278029 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:13.277992 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" event={"ID":"14afd400-a7ad-4787-8f96-1bde0831d319","Type":"ContainerStarted","Data":"11ecdd1ee5c1578ae6efa8c7a50a0e65cc4300eb944d4b144ca4332f12fc3d22"} Sep 13 20:10:13.293567 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:13.293516 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9h2f9" podStartSLOduration=1.855926684 podStartE2EDuration="6.293501474s" totalImagesPullingTime="4.43757479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:10:13.291815144 +0000 UTC m=+709.374960059" watchObservedRunningTime="2026-09-13 20:10:13.293501474 +0000 UTC m=+709.376646425" Sep 13 20:10:15.155728 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:15.155694 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:15.256622 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:15.256592 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:15.366387 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:15.366352 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:17.234236 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:17.234143 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:18.179208 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.179164 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7"] Sep 13 20:10:18.183680 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.183659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.198764 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.198740 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7"] Sep 13 20:10:18.310107 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.310067 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f299f7-6a08-401f-b612-db34371be96b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.310501 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.310136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdc5\" (UniqueName: \"kubernetes.io/projected/b8f299f7-6a08-401f-b612-db34371be96b-kube-api-access-xxdc5\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.411285 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.411244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdc5\" (UniqueName: \"kubernetes.io/projected/b8f299f7-6a08-401f-b612-db34371be96b-kube-api-access-xxdc5\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.411488 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.411373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f299f7-6a08-401f-b612-db34371be96b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.411785 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.411764 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f299f7-6a08-401f-b612-db34371be96b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.421577 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.421545 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdc5\" (UniqueName: \"kubernetes.io/projected/b8f299f7-6a08-401f-b612-db34371be96b-kube-api-access-xxdc5\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.493379 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.493296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:18.767676 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:18.767644 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7"] Sep 13 20:10:18.771061 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:10:18.771030 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f299f7_6a08_401f_b612_db34371be96b.slice/crio-6a1317497ed7eed625f99ef7250ee4e657d2095bcf42d7023a89b5d2fbcf0c0a WatchSource:0}: Error finding container 6a1317497ed7eed625f99ef7250ee4e657d2095bcf42d7023a89b5d2fbcf0c0a: Status 404 returned error can't find the container with id 6a1317497ed7eed625f99ef7250ee4e657d2095bcf42d7023a89b5d2fbcf0c0a Sep 13 20:10:19.198735 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.198651 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7"] Sep 13 20:10:19.204475 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.204450 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7"] Sep 13 20:10:19.217974 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.217948 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz"] Sep 13 20:10:19.218218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.218173 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" containerName="manager" containerID="cri-o://05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b" gracePeriod=2 Sep 13 20:10:19.219614 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.219591 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw"] Sep 13 20:10:19.219926 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.219915 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f299f7-6a08-401f-b612-db34371be96b" containerName="manager" Sep 13 20:10:19.219976 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.219928 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f299f7-6a08-401f-b612-db34371be96b" containerName="manager" Sep 13 20:10:19.220014 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.220001 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f299f7-6a08-401f-b612-db34371be96b" containerName="manager" Sep 13 20:10:19.223129 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.223111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.230978 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.230950 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz"] Sep 13 20:10:19.236764 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.236728 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e3f2f046-7380-4254-81d2-1e0938e90583\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\": pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.237357 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.237320 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw"] Sep 13 20:10:19.238702 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.238676 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.245070 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.245049 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf"] Sep 13 20:10:19.245456 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.245410 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" containerName="manager" Sep 13 20:10:19.245456 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.245454 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" containerName="manager" Sep 13 20:10:19.245588 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.245529 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" containerName="manager" Sep 13 20:10:19.248721 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.248704 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw"] Sep 13 20:10:19.248857 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.248838 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.248931 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.248851 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" containerName="manager" containerID="cri-o://7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337" gracePeriod=2 Sep 13 20:10:19.256148 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.255885 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw"] Sep 13 20:10:19.265575 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.265541 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf"] Sep 13 20:10:19.274212 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.274168 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst"] Sep 13 20:10:19.274544 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.274530 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" containerName="manager" Sep 13 20:10:19.274592 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.274546 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" containerName="manager" Sep 13 20:10:19.274631 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.274611 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" containerName="manager" Sep 13 20:10:19.277546 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.277528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" Sep 13 20:10:19.287201 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.287150 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:19.288073 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.288053 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst"] Sep 13 20:10:19.289358 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.289333 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.308303 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.308278 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" container="manager" Sep 13 20:10:19.312243 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.312211 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:19.314026 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.314003 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.315754 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.315730 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:19.317350 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.317320 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:19.318656 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.318638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whrg\" (UniqueName: \"kubernetes.io/projected/412ad084-d1c6-4e63-8ba3-993325adc335-kube-api-access-6whrg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dgqmf\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.318736 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.318668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/874e21fe-818e-4451-8d30-dcad3aeaeb20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-pxhpw\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.318736 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.318693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412ad084-d1c6-4e63-8ba3-993325adc335-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dgqmf\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.318736 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.318712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgjv\" (UniqueName: \"kubernetes.io/projected/874e21fe-818e-4451-8d30-dcad3aeaeb20-kube-api-access-tpgjv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-pxhpw\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.318910 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.318882 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.419877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.419788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whrg\" (UniqueName: \"kubernetes.io/projected/412ad084-d1c6-4e63-8ba3-993325adc335-kube-api-access-6whrg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dgqmf\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.419877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.419834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/874e21fe-818e-4451-8d30-dcad3aeaeb20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-pxhpw\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.419877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.419872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412ad084-d1c6-4e63-8ba3-993325adc335-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dgqmf\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.420294 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.419903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgjv\" (UniqueName: \"kubernetes.io/projected/874e21fe-818e-4451-8d30-dcad3aeaeb20-kube-api-access-tpgjv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-pxhpw\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.420294 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.419963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9x4\" (UniqueName: \"kubernetes.io/projected/fce9dc1c-61dd-4488-b0da-d2b636852f38-kube-api-access-7j9x4\") pod \"limitador-operator-controller-manager-74495c69f8-97jst\" (UID: \"fce9dc1c-61dd-4488-b0da-d2b636852f38\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" Sep 13 20:10:19.420635 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.420610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412ad084-d1c6-4e63-8ba3-993325adc335-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dgqmf\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.420866 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.420842 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/874e21fe-818e-4451-8d30-dcad3aeaeb20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-pxhpw\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.428595 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.428570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgjv\" (UniqueName: \"kubernetes.io/projected/874e21fe-818e-4451-8d30-dcad3aeaeb20-kube-api-access-tpgjv\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-pxhpw\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.429410 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.429353 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whrg\" (UniqueName: \"kubernetes.io/projected/412ad084-d1c6-4e63-8ba3-993325adc335-kube-api-access-6whrg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dgqmf\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.504844 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.504819 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.506818 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.506791 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:19.506915 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.506808 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:19.508499 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.508475 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:19.510213 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.510175 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.512009 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.511989 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:19.513759 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.513736 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:19.515360 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:19.515343 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:19.520744 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.520724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9x4\" (UniqueName: \"kubernetes.io/projected/fce9dc1c-61dd-4488-b0da-d2b636852f38-kube-api-access-7j9x4\") pod \"limitador-operator-controller-manager-74495c69f8-97jst\" (UID: \"fce9dc1c-61dd-4488-b0da-d2b636852f38\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" Sep 13 20:10:19.534936 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.534916 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9x4\" (UniqueName: \"kubernetes.io/projected/fce9dc1c-61dd-4488-b0da-d2b636852f38-kube-api-access-7j9x4\") pod \"limitador-operator-controller-manager-74495c69f8-97jst\" (UID: \"fce9dc1c-61dd-4488-b0da-d2b636852f38\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" Sep 13 20:10:19.536679 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.536662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:19.621435 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.621403 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zbqc\" (UniqueName: \"kubernetes.io/projected/e3f2f046-7380-4254-81d2-1e0938e90583-kube-api-access-9zbqc\") pod \"e3f2f046-7380-4254-81d2-1e0938e90583\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " Sep 13 20:10:19.621581 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.621485 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f2f046-7380-4254-81d2-1e0938e90583-extensions-socket-volume\") pod \"e3f2f046-7380-4254-81d2-1e0938e90583\" (UID: \"e3f2f046-7380-4254-81d2-1e0938e90583\") " Sep 13 20:10:19.621581 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.621556 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kgfv\" (UniqueName: \"kubernetes.io/projected/3810297a-f2bd-40c8-a684-4cd63aba83d8-kube-api-access-5kgfv\") pod \"3810297a-f2bd-40c8-a684-4cd63aba83d8\" (UID: \"3810297a-f2bd-40c8-a684-4cd63aba83d8\") " Sep 13 20:10:19.621965 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.621939 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f2f046-7380-4254-81d2-1e0938e90583-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e3f2f046-7380-4254-81d2-1e0938e90583" (UID: "e3f2f046-7380-4254-81d2-1e0938e90583"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 13 20:10:19.623571 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.623541 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f2f046-7380-4254-81d2-1e0938e90583-kube-api-access-9zbqc" (OuterVolumeSpecName: "kube-api-access-9zbqc") pod "e3f2f046-7380-4254-81d2-1e0938e90583" (UID: "e3f2f046-7380-4254-81d2-1e0938e90583"). InnerVolumeSpecName "kube-api-access-9zbqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:10:19.623678 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.623590 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3810297a-f2bd-40c8-a684-4cd63aba83d8-kube-api-access-5kgfv" (OuterVolumeSpecName: "kube-api-access-5kgfv") pod "3810297a-f2bd-40c8-a684-4cd63aba83d8" (UID: "3810297a-f2bd-40c8-a684-4cd63aba83d8"). InnerVolumeSpecName "kube-api-access-5kgfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:10:19.638770 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.638745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:19.643941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.643914 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" Sep 13 20:10:19.722466 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.722411 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kgfv\" (UniqueName: \"kubernetes.io/projected/3810297a-f2bd-40c8-a684-4cd63aba83d8-kube-api-access-5kgfv\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:19.722466 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.722467 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zbqc\" (UniqueName: \"kubernetes.io/projected/e3f2f046-7380-4254-81d2-1e0938e90583-kube-api-access-9zbqc\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:19.722660 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.722482 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f2f046-7380-4254-81d2-1e0938e90583-extensions-socket-volume\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:19.816811 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.816782 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw"] Sep 13 20:10:19.818655 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:10:19.818622 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874e21fe_818e_4451_8d30_dcad3aeaeb20.slice/crio-9b2842c9eecb59c0b42b1e7af55c8cd5a52120226d0f7f11bedff0ba4828c406 WatchSource:0}: Error finding container 9b2842c9eecb59c0b42b1e7af55c8cd5a52120226d0f7f11bedff0ba4828c406: Status 404 returned error can't find the container with id 9b2842c9eecb59c0b42b1e7af55c8cd5a52120226d0f7f11bedff0ba4828c406 Sep 13 20:10:19.926803 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.926772 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst"] Sep 13 20:10:19.929307 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:10:19.929275 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce9dc1c_61dd_4488_b0da_d2b636852f38.slice/crio-0bae3782c0a1662b93ed619f91b2304866bb26fc686ab94877680701f75d2387 WatchSource:0}: Error finding container 0bae3782c0a1662b93ed619f91b2304866bb26fc686ab94877680701f75d2387: Status 404 returned error can't find the container with id 0bae3782c0a1662b93ed619f91b2304866bb26fc686ab94877680701f75d2387 Sep 13 20:10:19.929922 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:19.929902 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf"] Sep 13 20:10:19.932155 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:10:19.932130 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412ad084_d1c6_4e63_8ba3_993325adc335.slice/crio-a019a8244634c37d650082c09f45db7284973cd8b945abaf6b7cf921775ab09e WatchSource:0}: Error finding container a019a8244634c37d650082c09f45db7284973cd8b945abaf6b7cf921775ab09e: Status 404 returned error can't find the container with id a019a8244634c37d650082c09f45db7284973cd8b945abaf6b7cf921775ab09e Sep 13 20:10:20.315405 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.315360 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" podUID="b8f299f7-6a08-401f-b612-db34371be96b" containerName="manager" containerID="cri-o://2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a" gracePeriod=2 Sep 13 20:10:20.317136 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.317103 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" event={"ID":"fce9dc1c-61dd-4488-b0da-d2b636852f38","Type":"ContainerStarted","Data":"58a0a2dcda117c5c6bc82a5cd639d862d14d69136ffddaf4bbe771d9eff199d7"} Sep 13 20:10:20.317313 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.317216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" event={"ID":"fce9dc1c-61dd-4488-b0da-d2b636852f38","Type":"ContainerStarted","Data":"0bae3782c0a1662b93ed619f91b2304866bb26fc686ab94877680701f75d2387"} Sep 13 20:10:20.317313 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.317252 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" Sep 13 20:10:20.317478 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.317354 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:20.318685 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.318661 2578 generic.go:356] "Generic (PLEG): container finished" podID="e3f2f046-7380-4254-81d2-1e0938e90583" containerID="05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b" exitCode=0 Sep 13 20:10:20.318798 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.318707 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:20.318798 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.318755 2578 scope.go:117] "RemoveContainer" containerID="05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b" Sep 13 20:10:20.319028 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.319003 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:20.322357 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.322333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" event={"ID":"412ad084-d1c6-4e63-8ba3-993325adc335","Type":"ContainerStarted","Data":"1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe"} Sep 13 20:10:20.322460 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.322328 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:20.322460 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.322382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" event={"ID":"412ad084-d1c6-4e63-8ba3-993325adc335","Type":"ContainerStarted","Data":"a019a8244634c37d650082c09f45db7284973cd8b945abaf6b7cf921775ab09e"} Sep 13 20:10:20.322831 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.322602 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:20.324493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.324466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" event={"ID":"874e21fe-818e-4451-8d30-dcad3aeaeb20","Type":"ContainerStarted","Data":"1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a"} Sep 13 20:10:20.324582 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.324515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" event={"ID":"874e21fe-818e-4451-8d30-dcad3aeaeb20","Type":"ContainerStarted","Data":"9b2842c9eecb59c0b42b1e7af55c8cd5a52120226d0f7f11bedff0ba4828c406"} Sep 13 20:10:20.324582 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.324542 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:20.325826 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.325808 2578 generic.go:356] "Generic (PLEG): container finished" podID="3810297a-f2bd-40c8-a684-4cd63aba83d8" containerID="7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337" exitCode=0 Sep 13 20:10:20.325898 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.325879 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:20.329274 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.329236 2578 scope.go:117] "RemoveContainer" containerID="05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b" Sep 13 20:10:20.329520 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.329502 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b\": container with ID starting with 05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b not found: ID does not exist" containerID="05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b" Sep 13 20:10:20.329569 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.329529 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b"} err="failed to get container status \"05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b\": rpc error: code = NotFound desc = could not find container \"05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b\": container with ID starting with 05026f5527432a82a7b67d554b5b00565ecfa7506cd8aa0476b00c380a17d37b not found: ID does not exist" Sep 13 20:10:20.329569 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.329546 2578 scope.go:117] "RemoveContainer" containerID="7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337" Sep 13 20:10:20.342106 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.342067 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" podStartSLOduration=1.3420507050000001 podStartE2EDuration="1.342050705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:10:20.341489574 +0000 UTC m=+716.424634510" watchObservedRunningTime="2026-09-13 20:10:20.342050705 +0000 UTC m=+716.425195620" Sep 13 20:10:20.343452 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.343425 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:20.345815 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.345789 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:20.347442 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.347424 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:20.372957 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.372901 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" podStartSLOduration=1.372885224 podStartE2EDuration="1.372885224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:10:20.370900854 +0000 UTC m=+716.454045770" watchObservedRunningTime="2026-09-13 20:10:20.372885224 +0000 UTC m=+716.456030138" Sep 13 20:10:20.373081 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.373031 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:20.375075 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.375043 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:20.396345 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.396315 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:20.396764 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.396725 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" podStartSLOduration=1.396709988 podStartE2EDuration="1.396709988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:10:20.394558971 +0000 UTC m=+716.477703885" watchObservedRunningTime="2026-09-13 20:10:20.396709988 +0000 UTC m=+716.479854902" Sep 13 20:10:20.397912 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.397890 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ttsqw\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ttsqw" Sep 13 20:10:20.399515 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.399491 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:20.401107 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.401087 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-jv7kz\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-jv7kz" Sep 13 20:10:20.420493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.420461 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3810297a-f2bd-40c8-a684-4cd63aba83d8" path="/var/lib/kubelet/pods/3810297a-f2bd-40c8-a684-4cd63aba83d8/volumes" Sep 13 20:10:20.420888 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.420867 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f2f046-7380-4254-81d2-1e0938e90583" path="/var/lib/kubelet/pods/e3f2f046-7380-4254-81d2-1e0938e90583/volumes" Sep 13 20:10:20.443447 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.443421 2578 scope.go:117] "RemoveContainer" containerID="7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337" Sep 13 20:10:20.443783 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:20.443762 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337\": container with ID starting with 7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337 not found: ID does not exist" containerID="7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337" Sep 13 20:10:20.443850 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.443791 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337"} err="failed to get container status \"7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337\": rpc error: code = NotFound desc = could not find container \"7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337\": container with ID starting with 7b8ecadedfa6c4ad51c8e2f57b261c8984899feab878bb0e5969a92c6dc0e337 not found: ID does not exist" Sep 13 20:10:20.563868 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.563834 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:20.629488 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.629410 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxdc5\" (UniqueName: \"kubernetes.io/projected/b8f299f7-6a08-401f-b612-db34371be96b-kube-api-access-xxdc5\") pod \"b8f299f7-6a08-401f-b612-db34371be96b\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " Sep 13 20:10:20.629626 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.629508 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f299f7-6a08-401f-b612-db34371be96b-extensions-socket-volume\") pod \"b8f299f7-6a08-401f-b612-db34371be96b\" (UID: \"b8f299f7-6a08-401f-b612-db34371be96b\") " Sep 13 20:10:20.629857 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.629832 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f299f7-6a08-401f-b612-db34371be96b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8f299f7-6a08-401f-b612-db34371be96b" (UID: "b8f299f7-6a08-401f-b612-db34371be96b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 13 20:10:20.631538 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.631521 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f299f7-6a08-401f-b612-db34371be96b-kube-api-access-xxdc5" (OuterVolumeSpecName: "kube-api-access-xxdc5") pod "b8f299f7-6a08-401f-b612-db34371be96b" (UID: "b8f299f7-6a08-401f-b612-db34371be96b"). InnerVolumeSpecName "kube-api-access-xxdc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:10:20.730877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.730839 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8f299f7-6a08-401f-b612-db34371be96b-extensions-socket-volume\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:20.730877 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:20.730870 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdc5\" (UniqueName: \"kubernetes.io/projected/b8f299f7-6a08-401f-b612-db34371be96b-kube-api-access-xxdc5\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:21.333617 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:21.333580 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8f299f7-6a08-401f-b612-db34371be96b" containerID="2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a" exitCode=0 Sep 13 20:10:21.334064 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:21.333624 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:21.334064 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:21.333685 2578 scope.go:117] "RemoveContainer" containerID="2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a" Sep 13 20:10:21.335780 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:21.335753 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:21.342966 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:21.342948 2578 scope.go:117] "RemoveContainer" containerID="2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a" Sep 13 20:10:21.343265 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:21.343247 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a\": container with ID starting with 2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a not found: ID does not exist" containerID="2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a" Sep 13 20:10:21.343310 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:21.343274 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a"} err="failed to get container status \"2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a\": rpc error: code = NotFound desc = could not find container \"2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a\": container with ID starting with 2a6d77913448ae4c646ebbb6de16731ec97bdf94c9b98eb928ffd56f98156e1a not found: ID does not exist" Sep 13 20:10:21.344263 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:21.344243 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h6nt7\" is forbidden: User \"system:node:ip-10-0-141-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-146.ec2.internal' and this object" podUID="b8f299f7-6a08-401f-b612-db34371be96b" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h6nt7" Sep 13 20:10:21.806912 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:21.806877 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:21.909616 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:21.909582 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:22.419496 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:22.419457 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f299f7-6a08-401f-b612-db34371be96b" path="/var/lib/kubelet/pods/b8f299f7-6a08-401f-b612-db34371be96b/volumes" Sep 13 20:10:24.779869 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:24.779834 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:24.880845 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:24.880811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:30.576699 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:30.576662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:30.629262 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:30.629221 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:31.180218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.180168 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:31.230954 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.230924 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:31.337452 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.337422 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-97jst" Sep 13 20:10:31.337634 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.337492 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:31.337634 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.337622 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:31.429430 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.429401 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf"] Sep 13 20:10:31.429608 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.429578 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" podUID="412ad084-d1c6-4e63-8ba3-993325adc335" containerName="manager" containerID="cri-o://1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe" gracePeriod=10 Sep 13 20:10:31.675013 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.674989 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:31.798568 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.798531 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg"] Sep 13 20:10:31.798891 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.798879 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412ad084-d1c6-4e63-8ba3-993325adc335" containerName="manager" Sep 13 20:10:31.798934 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.798893 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="412ad084-d1c6-4e63-8ba3-993325adc335" containerName="manager" Sep 13 20:10:31.798967 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.798952 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="412ad084-d1c6-4e63-8ba3-993325adc335" containerName="manager" Sep 13 20:10:31.801628 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.801599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:31.817976 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.817955 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg"] Sep 13 20:10:31.822156 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.822137 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6whrg\" (UniqueName: \"kubernetes.io/projected/412ad084-d1c6-4e63-8ba3-993325adc335-kube-api-access-6whrg\") pod \"412ad084-d1c6-4e63-8ba3-993325adc335\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " Sep 13 20:10:31.822273 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.822165 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412ad084-d1c6-4e63-8ba3-993325adc335-extensions-socket-volume\") pod \"412ad084-d1c6-4e63-8ba3-993325adc335\" (UID: \"412ad084-d1c6-4e63-8ba3-993325adc335\") " Sep 13 20:10:31.822563 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.822539 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412ad084-d1c6-4e63-8ba3-993325adc335-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "412ad084-d1c6-4e63-8ba3-993325adc335" (UID: "412ad084-d1c6-4e63-8ba3-993325adc335"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 13 20:10:31.824226 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.824203 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412ad084-d1c6-4e63-8ba3-993325adc335-kube-api-access-6whrg" (OuterVolumeSpecName: "kube-api-access-6whrg") pod "412ad084-d1c6-4e63-8ba3-993325adc335" (UID: "412ad084-d1c6-4e63-8ba3-993325adc335"). InnerVolumeSpecName "kube-api-access-6whrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:10:31.923532 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.923491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cbf8a7d-12f1-4f77-9081-5897b71a62eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hhvtg\" (UID: \"8cbf8a7d-12f1-4f77-9081-5897b71a62eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:31.923702 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.923555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkrs\" (UniqueName: \"kubernetes.io/projected/8cbf8a7d-12f1-4f77-9081-5897b71a62eb-kube-api-access-trkrs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hhvtg\" (UID: \"8cbf8a7d-12f1-4f77-9081-5897b71a62eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:31.923702 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.923672 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whrg\" (UniqueName: \"kubernetes.io/projected/412ad084-d1c6-4e63-8ba3-993325adc335-kube-api-access-6whrg\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:31.923702 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:31.923693 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/412ad084-d1c6-4e63-8ba3-993325adc335-extensions-socket-volume\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:32.024522 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.024481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cbf8a7d-12f1-4f77-9081-5897b71a62eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hhvtg\" (UID: \"8cbf8a7d-12f1-4f77-9081-5897b71a62eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:32.024522 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.024530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkrs\" (UniqueName: \"kubernetes.io/projected/8cbf8a7d-12f1-4f77-9081-5897b71a62eb-kube-api-access-trkrs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hhvtg\" (UID: \"8cbf8a7d-12f1-4f77-9081-5897b71a62eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:32.024862 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.024841 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cbf8a7d-12f1-4f77-9081-5897b71a62eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hhvtg\" (UID: \"8cbf8a7d-12f1-4f77-9081-5897b71a62eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:32.032777 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.032744 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkrs\" (UniqueName: \"kubernetes.io/projected/8cbf8a7d-12f1-4f77-9081-5897b71a62eb-kube-api-access-trkrs\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hhvtg\" (UID: \"8cbf8a7d-12f1-4f77-9081-5897b71a62eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:32.111831 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.111732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:32.390950 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.390921 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg"] Sep 13 20:10:32.392291 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.392261 2578 generic.go:356] "Generic (PLEG): container finished" podID="412ad084-d1c6-4e63-8ba3-993325adc335" containerID="1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe" exitCode=0 Sep 13 20:10:32.392392 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.392296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" event={"ID":"412ad084-d1c6-4e63-8ba3-993325adc335","Type":"ContainerDied","Data":"1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe"} Sep 13 20:10:32.392392 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.392311 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" Sep 13 20:10:32.392392 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.392349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf" event={"ID":"412ad084-d1c6-4e63-8ba3-993325adc335","Type":"ContainerDied","Data":"a019a8244634c37d650082c09f45db7284973cd8b945abaf6b7cf921775ab09e"} Sep 13 20:10:32.392392 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.392373 2578 scope.go:117] "RemoveContainer" containerID="1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe" Sep 13 20:10:32.392953 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:10:32.392925 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbf8a7d_12f1_4f77_9081_5897b71a62eb.slice/crio-658654b2654bac3508ccad0ca7b4d1aae700b3e33a4a6b341fec600842b8ae47 WatchSource:0}: Error finding container 658654b2654bac3508ccad0ca7b4d1aae700b3e33a4a6b341fec600842b8ae47: Status 404 returned error can't find the container with id 658654b2654bac3508ccad0ca7b4d1aae700b3e33a4a6b341fec600842b8ae47 Sep 13 20:10:32.400860 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.400841 2578 scope.go:117] "RemoveContainer" containerID="1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe" Sep 13 20:10:32.401128 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:32.401110 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe\": container with ID starting with 1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe not found: ID does not exist" containerID="1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe" Sep 13 20:10:32.401222 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.401135 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe"} err="failed to get container status \"1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe\": rpc error: code = NotFound desc = could not find container \"1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe\": container with ID starting with 1820f833dec57ec4110bc1b8c002265a822dac591bbb5cc7ad47b6c8b78e9bbe not found: ID does not exist" Sep 13 20:10:32.421263 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.421240 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf"] Sep 13 20:10:32.426489 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:32.426470 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dgqmf"] Sep 13 20:10:33.400711 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:33.400675 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" event={"ID":"8cbf8a7d-12f1-4f77-9081-5897b71a62eb","Type":"ContainerStarted","Data":"d3b9b5a536cb19f6dcdf605cb9b5609a8472dfeca94655ca07470d5684b1ea1f"} Sep 13 20:10:33.401346 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:33.400725 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" event={"ID":"8cbf8a7d-12f1-4f77-9081-5897b71a62eb","Type":"ContainerStarted","Data":"658654b2654bac3508ccad0ca7b4d1aae700b3e33a4a6b341fec600842b8ae47"} Sep 13 20:10:33.401346 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:33.400790 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:33.420924 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:33.420882 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" podStartSLOduration=2.420870424 podStartE2EDuration="2.420870424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:10:33.418306373 +0000 UTC m=+729.501451283" watchObservedRunningTime="2026-09-13 20:10:33.420870424 +0000 UTC m=+729.504015337" Sep 13 20:10:34.424576 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:34.424545 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412ad084-d1c6-4e63-8ba3-993325adc335" path="/var/lib/kubelet/pods/412ad084-d1c6-4e63-8ba3-993325adc335/volumes" Sep 13 20:10:34.634437 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:34.634402 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:34.735804 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:34.735720 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:34.834312 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:34.834261 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:44.408922 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:44.408885 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hhvtg" Sep 13 20:10:44.454271 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:44.454236 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw"] Sep 13 20:10:44.454733 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:44.454698 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" podUID="874e21fe-818e-4451-8d30-dcad3aeaeb20" containerName="manager" containerID="cri-o://1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a" gracePeriod=10 Sep 13 20:10:45.235345 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.235313 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:45.334952 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.334869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/874e21fe-818e-4451-8d30-dcad3aeaeb20-extensions-socket-volume\") pod \"874e21fe-818e-4451-8d30-dcad3aeaeb20\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " Sep 13 20:10:45.334952 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.334956 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpgjv\" (UniqueName: \"kubernetes.io/projected/874e21fe-818e-4451-8d30-dcad3aeaeb20-kube-api-access-tpgjv\") pod \"874e21fe-818e-4451-8d30-dcad3aeaeb20\" (UID: \"874e21fe-818e-4451-8d30-dcad3aeaeb20\") " Sep 13 20:10:45.335245 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.335202 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874e21fe-818e-4451-8d30-dcad3aeaeb20-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "874e21fe-818e-4451-8d30-dcad3aeaeb20" (UID: "874e21fe-818e-4451-8d30-dcad3aeaeb20"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 13 20:10:45.336949 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.336931 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874e21fe-818e-4451-8d30-dcad3aeaeb20-kube-api-access-tpgjv" (OuterVolumeSpecName: "kube-api-access-tpgjv") pod "874e21fe-818e-4451-8d30-dcad3aeaeb20" (UID: "874e21fe-818e-4451-8d30-dcad3aeaeb20"). InnerVolumeSpecName "kube-api-access-tpgjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:10:45.436212 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.436148 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/874e21fe-818e-4451-8d30-dcad3aeaeb20-extensions-socket-volume\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:45.436212 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.436203 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpgjv\" (UniqueName: \"kubernetes.io/projected/874e21fe-818e-4451-8d30-dcad3aeaeb20-kube-api-access-tpgjv\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:10:45.460875 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.460836 2578 generic.go:356] "Generic (PLEG): container finished" podID="874e21fe-818e-4451-8d30-dcad3aeaeb20" containerID="1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a" exitCode=0 Sep 13 20:10:45.461055 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.460879 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" Sep 13 20:10:45.461055 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.460925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" event={"ID":"874e21fe-818e-4451-8d30-dcad3aeaeb20","Type":"ContainerDied","Data":"1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a"} Sep 13 20:10:45.461055 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.460976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw" event={"ID":"874e21fe-818e-4451-8d30-dcad3aeaeb20","Type":"ContainerDied","Data":"9b2842c9eecb59c0b42b1e7af55c8cd5a52120226d0f7f11bedff0ba4828c406"} Sep 13 20:10:45.461055 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.460999 2578 scope.go:117] "RemoveContainer" containerID="1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a" Sep 13 20:10:45.472268 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.472247 2578 scope.go:117] "RemoveContainer" containerID="1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a" Sep 13 20:10:45.472518 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:10:45.472497 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a\": container with ID starting with 1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a not found: ID does not exist" containerID="1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a" Sep 13 20:10:45.472566 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.472528 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a"} err="failed to get container status \"1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a\": rpc error: code = NotFound desc = could not find container \"1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a\": container with ID starting with 1a4046b63970d7502604bc4e0a1198a4726df6fe7e81cf598e98dc37841da37a not found: ID does not exist" Sep 13 20:10:45.483090 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.483064 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw"] Sep 13 20:10:45.494774 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:45.494749 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-pxhpw"] Sep 13 20:10:46.196593 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:46.196559 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:46.246043 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:46.246007 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:46.420487 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:46.420457 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="874e21fe-818e-4451-8d30-dcad3aeaeb20" path="/var/lib/kubelet/pods/874e21fe-818e-4451-8d30-dcad3aeaeb20/volumes" Sep 13 20:10:47.715374 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:47.715339 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:47.834596 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:47.834562 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:10:57.799476 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:10:57.799436 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:00.786812 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.786776 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:00.787176 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.786840 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rjcdm"] Sep 13 20:11:00.787273 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.787259 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="874e21fe-818e-4451-8d30-dcad3aeaeb20" containerName="manager" Sep 13 20:11:00.787324 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.787275 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="874e21fe-818e-4451-8d30-dcad3aeaeb20" containerName="manager" Sep 13 20:11:00.787361 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.787338 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="874e21fe-818e-4451-8d30-dcad3aeaeb20" containerName="manager" Sep 13 20:11:00.790575 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.790552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:00.792547 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.792529 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 13 20:11:00.796124 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.796103 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rjcdm"] Sep 13 20:11:00.863762 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.863726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9vnr\" (UniqueName: \"kubernetes.io/projected/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-kube-api-access-w9vnr\") pod \"limitador-limitador-8588749977-rjcdm\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:00.863954 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.863771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-config-file\") pod \"limitador-limitador-8588749977-rjcdm\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:00.883218 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.883169 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:00.888603 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.888578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rjcdm"] Sep 13 20:11:00.964654 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.964617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9vnr\" (UniqueName: \"kubernetes.io/projected/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-kube-api-access-w9vnr\") pod \"limitador-limitador-8588749977-rjcdm\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:00.964834 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.964667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-config-file\") pod \"limitador-limitador-8588749977-rjcdm\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:00.965350 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.965331 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-config-file\") pod \"limitador-limitador-8588749977-rjcdm\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:00.971999 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.971975 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9vnr\" (UniqueName: \"kubernetes.io/projected/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-kube-api-access-w9vnr\") pod \"limitador-limitador-8588749977-rjcdm\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:00.993899 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:00.993867 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:01.101409 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.101320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:01.374833 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.374805 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rjcdm"] Sep 13 20:11:01.376751 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:11:01.376717 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8bbff6_2e2a_444e_b7a1_3ae0c175664a.slice/crio-4e43932535669812cc863ee8ec848fef6b6c08bae860761c302d2bee461fe267 WatchSource:0}: Error finding container 4e43932535669812cc863ee8ec848fef6b6c08bae860761c302d2bee461fe267: Status 404 returned error can't find the container with id 4e43932535669812cc863ee8ec848fef6b6c08bae860761c302d2bee461fe267 Sep 13 20:11:01.547173 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.547130 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" event={"ID":"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a","Type":"ContainerStarted","Data":"4e43932535669812cc863ee8ec848fef6b6c08bae860761c302d2bee461fe267"} Sep 13 20:11:01.564270 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.564235 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s4bcq"] Sep 13 20:11:01.568858 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.568837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" Sep 13 20:11:01.570923 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.570900 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-prs4z\"" Sep 13 20:11:01.574489 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.574466 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s4bcq"] Sep 13 20:11:01.596391 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.596362 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:01.670103 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.669990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4tpr\" (UniqueName: \"kubernetes.io/projected/a46fa247-c8e7-4702-82b0-41f7af82a298-kube-api-access-k4tpr\") pod \"authorino-7bcfd8f9cd-s4bcq\" (UID: \"a46fa247-c8e7-4702-82b0-41f7af82a298\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" Sep 13 20:11:01.698005 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.697971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:01.771305 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.771262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tpr\" (UniqueName: \"kubernetes.io/projected/a46fa247-c8e7-4702-82b0-41f7af82a298-kube-api-access-k4tpr\") pod \"authorino-7bcfd8f9cd-s4bcq\" (UID: \"a46fa247-c8e7-4702-82b0-41f7af82a298\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" Sep 13 20:11:01.778831 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.778773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tpr\" (UniqueName: \"kubernetes.io/projected/a46fa247-c8e7-4702-82b0-41f7af82a298-kube-api-access-k4tpr\") pod \"authorino-7bcfd8f9cd-s4bcq\" (UID: \"a46fa247-c8e7-4702-82b0-41f7af82a298\") " pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" Sep 13 20:11:01.880076 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:01.880037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" Sep 13 20:11:02.147114 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:02.147089 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s4bcq"] Sep 13 20:11:02.148662 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:11:02.148631 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda46fa247_c8e7_4702_82b0_41f7af82a298.slice/crio-0871b5ae4120593015c65aae67145e96aebdc209bde7e4c7b80d1db2bfe4056c WatchSource:0}: Error finding container 0871b5ae4120593015c65aae67145e96aebdc209bde7e4c7b80d1db2bfe4056c: Status 404 returned error can't find the container with id 0871b5ae4120593015c65aae67145e96aebdc209bde7e4c7b80d1db2bfe4056c Sep 13 20:11:02.244738 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:02.244702 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:02.298552 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:02.298520 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:02.553777 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:02.553745 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" event={"ID":"a46fa247-c8e7-4702-82b0-41f7af82a298","Type":"ContainerStarted","Data":"0871b5ae4120593015c65aae67145e96aebdc209bde7e4c7b80d1db2bfe4056c"} Sep 13 20:11:03.560910 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:03.560872 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" event={"ID":"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a","Type":"ContainerStarted","Data":"faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3"} Sep 13 20:11:03.561381 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:03.560996 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:03.575956 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:03.575905 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" podStartSLOduration=2.246354838 podStartE2EDuration="3.57588719s" totalImagesPullingTime="1.329532352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:11:03.574547575 +0000 UTC m=+759.657692490" watchObservedRunningTime="2026-09-13 20:11:03.57588719 +0000 UTC m=+759.659032101" Sep 13 20:11:04.567045 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:04.567007 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" event={"ID":"a46fa247-c8e7-4702-82b0-41f7af82a298","Type":"ContainerStarted","Data":"735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e"} Sep 13 20:11:04.581147 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:04.581096 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" podStartSLOduration=1.849892059 podStartE2EDuration="3.58108122s" totalImagesPullingTime="1.731189161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:11:04.579980794 +0000 UTC m=+760.663125706" watchObservedRunningTime="2026-09-13 20:11:04.58108122 +0000 UTC m=+760.664226133" Sep 13 20:11:05.406341 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:05.406302 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s4bcq"] Sep 13 20:11:06.577162 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:06.577124 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" podUID="a46fa247-c8e7-4702-82b0-41f7af82a298" containerName="authorino" containerID="cri-o://735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e" gracePeriod=30 Sep 13 20:11:06.813151 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:06.813120 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" Sep 13 20:11:06.914352 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:06.914259 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4tpr\" (UniqueName: \"kubernetes.io/projected/a46fa247-c8e7-4702-82b0-41f7af82a298-kube-api-access-k4tpr\") pod \"a46fa247-c8e7-4702-82b0-41f7af82a298\" (UID: \"a46fa247-c8e7-4702-82b0-41f7af82a298\") " Sep 13 20:11:06.916211 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:06.916168 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46fa247-c8e7-4702-82b0-41f7af82a298-kube-api-access-k4tpr" (OuterVolumeSpecName: "kube-api-access-k4tpr") pod "a46fa247-c8e7-4702-82b0-41f7af82a298" (UID: "a46fa247-c8e7-4702-82b0-41f7af82a298"). InnerVolumeSpecName "kube-api-access-k4tpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:11:07.015641 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.015610 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4tpr\" (UniqueName: \"kubernetes.io/projected/a46fa247-c8e7-4702-82b0-41f7af82a298-kube-api-access-k4tpr\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:11:07.585493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.585456 2578 generic.go:356] "Generic (PLEG): container finished" podID="a46fa247-c8e7-4702-82b0-41f7af82a298" containerID="735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e" exitCode=0 Sep 13 20:11:07.585493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.585492 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" Sep 13 20:11:07.585984 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.585493 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" event={"ID":"a46fa247-c8e7-4702-82b0-41f7af82a298","Type":"ContainerDied","Data":"735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e"} Sep 13 20:11:07.585984 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.585546 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-s4bcq" event={"ID":"a46fa247-c8e7-4702-82b0-41f7af82a298","Type":"ContainerDied","Data":"0871b5ae4120593015c65aae67145e96aebdc209bde7e4c7b80d1db2bfe4056c"} Sep 13 20:11:07.585984 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.585563 2578 scope.go:117] "RemoveContainer" containerID="735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e" Sep 13 20:11:07.594397 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.594381 2578 scope.go:117] "RemoveContainer" containerID="735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e" Sep 13 20:11:07.594664 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:11:07.594644 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e\": container with ID starting with 735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e not found: ID does not exist" containerID="735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e" Sep 13 20:11:07.594704 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.594674 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e"} err="failed to get container status \"735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e\": rpc error: code = NotFound desc = could not find container \"735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e\": container with ID starting with 735b6ead296ef90e3802817f0996fb04c8755335269eab4f635cc318f0cbf58e not found: ID does not exist" Sep 13 20:11:07.608269 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.606138 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s4bcq"] Sep 13 20:11:07.610954 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:07.610160 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-s4bcq"] Sep 13 20:11:08.420108 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:08.420076 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46fa247-c8e7-4702-82b0-41f7af82a298" path="/var/lib/kubelet/pods/a46fa247-c8e7-4702-82b0-41f7af82a298/volumes" Sep 13 20:11:14.568377 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:14.568345 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:15.406698 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:15.406663 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rjcdm"] Sep 13 20:11:15.406888 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:15.406840 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" podUID="7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" containerName="limitador" containerID="cri-o://faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3" gracePeriod=30 Sep 13 20:11:15.951039 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:15.951012 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:16.094341 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.094303 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9vnr\" (UniqueName: \"kubernetes.io/projected/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-kube-api-access-w9vnr\") pod \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " Sep 13 20:11:16.094500 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.094397 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-config-file\") pod \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\" (UID: \"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a\") " Sep 13 20:11:16.094776 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.094751 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-config-file" (OuterVolumeSpecName: "config-file") pod "7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" (UID: "7d8bbff6-2e2a-444e-b7a1-3ae0c175664a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 13 20:11:16.096506 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.096485 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-kube-api-access-w9vnr" (OuterVolumeSpecName: "kube-api-access-w9vnr") pod "7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" (UID: "7d8bbff6-2e2a-444e-b7a1-3ae0c175664a"). InnerVolumeSpecName "kube-api-access-w9vnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:11:16.195272 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.195234 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-config-file\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:11:16.195272 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.195266 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9vnr\" (UniqueName: \"kubernetes.io/projected/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a-kube-api-access-w9vnr\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:11:16.630934 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.630893 2578 generic.go:356] "Generic (PLEG): container finished" podID="7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" containerID="faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3" exitCode=0 Sep 13 20:11:16.631091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.630940 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" Sep 13 20:11:16.631091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.630979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" event={"ID":"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a","Type":"ContainerDied","Data":"faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3"} Sep 13 20:11:16.631091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.631032 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-rjcdm" event={"ID":"7d8bbff6-2e2a-444e-b7a1-3ae0c175664a","Type":"ContainerDied","Data":"4e43932535669812cc863ee8ec848fef6b6c08bae860761c302d2bee461fe267"} Sep 13 20:11:16.631091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.631053 2578 scope.go:117] "RemoveContainer" containerID="faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3" Sep 13 20:11:16.639280 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.639255 2578 scope.go:117] "RemoveContainer" containerID="faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3" Sep 13 20:11:16.639510 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:11:16.639490 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3\": container with ID starting with faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3 not found: ID does not exist" containerID="faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3" Sep 13 20:11:16.639577 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.639521 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3"} err="failed to get container status \"faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3\": rpc error: code = NotFound desc = could not find container \"faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3\": container with ID starting with faad6e0e1a1232eedfefc31e5de85574c132da3d76b28efe0b15deccd829ebc3 not found: ID does not exist" Sep 13 20:11:16.644758 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.644733 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rjcdm"] Sep 13 20:11:16.647048 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:16.647024 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-rjcdm"] Sep 13 20:11:18.420135 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:18.420102 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" path="/var/lib/kubelet/pods/7d8bbff6-2e2a-444e-b7a1-3ae0c175664a/volumes" Sep 13 20:11:20.780734 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:20.780689 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:20.880030 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:20.879995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:21.219423 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.219347 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh"] Sep 13 20:11:21.219897 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.219881 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" containerName="limitador" Sep 13 20:11:21.219941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.219900 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" containerName="limitador" Sep 13 20:11:21.219941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.219925 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46fa247-c8e7-4702-82b0-41f7af82a298" containerName="authorino" Sep 13 20:11:21.219941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.219934 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46fa247-c8e7-4702-82b0-41f7af82a298" containerName="authorino" Sep 13 20:11:21.220036 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.220019 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46fa247-c8e7-4702-82b0-41f7af82a298" containerName="authorino" Sep 13 20:11:21.220036 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.220030 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d8bbff6-2e2a-444e-b7a1-3ae0c175664a" containerName="limitador" Sep 13 20:11:21.223338 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.223316 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.225696 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.225666 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 13 20:11:21.225840 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.225818 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 13 20:11:21.226546 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.226523 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6d4kd\"" Sep 13 20:11:21.226622 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.226548 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 13 20:11:21.232328 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.232306 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh"] Sep 13 20:11:21.254263 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.254230 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:21.339732 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.339700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnjgd\" (UniqueName: \"kubernetes.io/projected/cc520c47-37eb-49cf-a695-fe9dda4ad946-kube-api-access-dnjgd\") pod \"postgres-69fb4594cf-bbjnh\" (UID: \"cc520c47-37eb-49cf-a695-fe9dda4ad946\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.339924 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.339747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc520c47-37eb-49cf-a695-fe9dda4ad946-data\") pod \"postgres-69fb4594cf-bbjnh\" (UID: \"cc520c47-37eb-49cf-a695-fe9dda4ad946\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.440288 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.440245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnjgd\" (UniqueName: \"kubernetes.io/projected/cc520c47-37eb-49cf-a695-fe9dda4ad946-kube-api-access-dnjgd\") pod \"postgres-69fb4594cf-bbjnh\" (UID: \"cc520c47-37eb-49cf-a695-fe9dda4ad946\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.440497 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.440308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc520c47-37eb-49cf-a695-fe9dda4ad946-data\") pod \"postgres-69fb4594cf-bbjnh\" (UID: \"cc520c47-37eb-49cf-a695-fe9dda4ad946\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.440709 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.440689 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc520c47-37eb-49cf-a695-fe9dda4ad946-data\") pod \"postgres-69fb4594cf-bbjnh\" (UID: \"cc520c47-37eb-49cf-a695-fe9dda4ad946\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.447959 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.447936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnjgd\" (UniqueName: \"kubernetes.io/projected/cc520c47-37eb-49cf-a695-fe9dda4ad946-kube-api-access-dnjgd\") pod \"postgres-69fb4594cf-bbjnh\" (UID: \"cc520c47-37eb-49cf-a695-fe9dda4ad946\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.535637 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.535606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:21.803220 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:21.803174 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh"] Sep 13 20:11:21.805598 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:11:21.805568 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc520c47_37eb_49cf_a695_fe9dda4ad946.slice/crio-2a5ecf628c3ac48b4bc4a6c93095015ddf5fbe9e770e27c35093b3561654f476 WatchSource:0}: Error finding container 2a5ecf628c3ac48b4bc4a6c93095015ddf5fbe9e770e27c35093b3561654f476: Status 404 returned error can't find the container with id 2a5ecf628c3ac48b4bc4a6c93095015ddf5fbe9e770e27c35093b3561654f476 Sep 13 20:11:22.663698 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:22.663665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" event={"ID":"cc520c47-37eb-49cf-a695-fe9dda4ad946","Type":"ContainerStarted","Data":"2a5ecf628c3ac48b4bc4a6c93095015ddf5fbe9e770e27c35093b3561654f476"} Sep 13 20:11:24.676387 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:24.676303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" event={"ID":"cc520c47-37eb-49cf-a695-fe9dda4ad946","Type":"ContainerStarted","Data":"e5617948c4ba5365aa3ded25feee31326968988ad929766a1736382d58c3ebfa"} Sep 13 20:11:24.676745 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:24.676553 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:24.695503 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:24.695448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" podStartSLOduration=1.435843323 podStartE2EDuration="3.695433432s" totalImagesPullingTime="2.259590109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:11:24.693852697 +0000 UTC m=+780.776997623" watchObservedRunningTime="2026-09-13 20:11:24.695433432 +0000 UTC m=+780.778578361" Sep 13 20:11:30.714228 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:30.714196 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bbjnh" Sep 13 20:11:31.000458 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.000325 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:31.148315 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.148272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:31.208293 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.208256 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-qlt8w"] Sep 13 20:11:31.214748 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.214725 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qlt8w" Sep 13 20:11:31.216894 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.216872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-prs4z\"" Sep 13 20:11:31.217469 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.217446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-qlt8w"] Sep 13 20:11:31.260935 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.260901 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qlt8w"] Sep 13 20:11:31.261165 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:11:31.261145 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nhs7f], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-qlt8w" podUID="006d4016-cb97-4751-9c1b-83034e899d0e" Sep 13 20:11:31.284196 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.284146 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mgnbf"] Sep 13 20:11:31.288656 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.288630 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.291023 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.290993 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 13 20:11:31.291147 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.291073 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mgnbf"] Sep 13 20:11:31.328594 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.328560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhs7f\" (UniqueName: \"kubernetes.io/projected/006d4016-cb97-4751-9c1b-83034e899d0e-kube-api-access-nhs7f\") pod \"authorino-557f876f69-qlt8w\" (UID: \"006d4016-cb97-4751-9c1b-83034e899d0e\") " pod="kuadrant-system/authorino-557f876f69-qlt8w" Sep 13 20:11:31.429588 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.429548 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mgnbf"] Sep 13 20:11:31.429803 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:11:31.429783 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zm5mn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" podUID="faa57dad-5276-490f-8aa0-dea7ffe25b6c" Sep 13 20:11:31.429848 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.429812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa57dad-5276-490f-8aa0-dea7ffe25b6c-tls-cert\") pod \"authorino-6d4dfc649f-mgnbf\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.429888 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.429845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5mn\" (UniqueName: \"kubernetes.io/projected/faa57dad-5276-490f-8aa0-dea7ffe25b6c-kube-api-access-zm5mn\") pod \"authorino-6d4dfc649f-mgnbf\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.429944 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.429928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhs7f\" (UniqueName: \"kubernetes.io/projected/006d4016-cb97-4751-9c1b-83034e899d0e-kube-api-access-nhs7f\") pod \"authorino-557f876f69-qlt8w\" (UID: \"006d4016-cb97-4751-9c1b-83034e899d0e\") " pod="kuadrant-system/authorino-557f876f69-qlt8w" Sep 13 20:11:31.444270 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.444245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhs7f\" (UniqueName: \"kubernetes.io/projected/006d4016-cb97-4751-9c1b-83034e899d0e-kube-api-access-nhs7f\") pod \"authorino-557f876f69-qlt8w\" (UID: \"006d4016-cb97-4751-9c1b-83034e899d0e\") " pod="kuadrant-system/authorino-557f876f69-qlt8w" Sep 13 20:11:31.531039 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.530954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa57dad-5276-490f-8aa0-dea7ffe25b6c-tls-cert\") pod \"authorino-6d4dfc649f-mgnbf\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.531039 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.531001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5mn\" (UniqueName: \"kubernetes.io/projected/faa57dad-5276-490f-8aa0-dea7ffe25b6c-kube-api-access-zm5mn\") pod \"authorino-6d4dfc649f-mgnbf\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.533607 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.533581 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa57dad-5276-490f-8aa0-dea7ffe25b6c-tls-cert\") pod \"authorino-6d4dfc649f-mgnbf\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.541290 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.541269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5mn\" (UniqueName: \"kubernetes.io/projected/faa57dad-5276-490f-8aa0-dea7ffe25b6c-kube-api-access-zm5mn\") pod \"authorino-6d4dfc649f-mgnbf\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.709249 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.709221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qlt8w" Sep 13 20:11:31.709409 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.709221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.717659 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.717638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qlt8w" Sep 13 20:11:31.720470 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.720454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:31.833637 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.833545 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm5mn\" (UniqueName: \"kubernetes.io/projected/faa57dad-5276-490f-8aa0-dea7ffe25b6c-kube-api-access-zm5mn\") pod \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " Sep 13 20:11:31.833637 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.833608 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhs7f\" (UniqueName: \"kubernetes.io/projected/006d4016-cb97-4751-9c1b-83034e899d0e-kube-api-access-nhs7f\") pod \"006d4016-cb97-4751-9c1b-83034e899d0e\" (UID: \"006d4016-cb97-4751-9c1b-83034e899d0e\") " Sep 13 20:11:31.833865 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.833654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa57dad-5276-490f-8aa0-dea7ffe25b6c-tls-cert\") pod \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\" (UID: \"faa57dad-5276-490f-8aa0-dea7ffe25b6c\") " Sep 13 20:11:31.835752 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.835723 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa57dad-5276-490f-8aa0-dea7ffe25b6c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "faa57dad-5276-490f-8aa0-dea7ffe25b6c" (UID: "faa57dad-5276-490f-8aa0-dea7ffe25b6c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 13 20:11:31.835862 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.835839 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006d4016-cb97-4751-9c1b-83034e899d0e-kube-api-access-nhs7f" (OuterVolumeSpecName: "kube-api-access-nhs7f") pod "006d4016-cb97-4751-9c1b-83034e899d0e" (UID: "006d4016-cb97-4751-9c1b-83034e899d0e"). InnerVolumeSpecName "kube-api-access-nhs7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:11:31.835907 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.835869 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa57dad-5276-490f-8aa0-dea7ffe25b6c-kube-api-access-zm5mn" (OuterVolumeSpecName: "kube-api-access-zm5mn") pod "faa57dad-5276-490f-8aa0-dea7ffe25b6c" (UID: "faa57dad-5276-490f-8aa0-dea7ffe25b6c"). InnerVolumeSpecName "kube-api-access-zm5mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:11:31.935030 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.934996 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5mn\" (UniqueName: \"kubernetes.io/projected/faa57dad-5276-490f-8aa0-dea7ffe25b6c-kube-api-access-zm5mn\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:11:31.935030 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.935025 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhs7f\" (UniqueName: \"kubernetes.io/projected/006d4016-cb97-4751-9c1b-83034e899d0e-kube-api-access-nhs7f\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:11:31.935030 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:31.935035 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/faa57dad-5276-490f-8aa0-dea7ffe25b6c-tls-cert\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:11:32.714628 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:32.714558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-qlt8w" Sep 13 20:11:32.714771 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:32.714556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mgnbf" Sep 13 20:11:32.742578 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:32.742548 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-qlt8w"] Sep 13 20:11:32.744145 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:32.744123 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-qlt8w"] Sep 13 20:11:32.762822 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:32.762788 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mgnbf"] Sep 13 20:11:32.764366 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:32.764344 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mgnbf"] Sep 13 20:11:34.427731 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:34.427696 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="006d4016-cb97-4751-9c1b-83034e899d0e" path="/var/lib/kubelet/pods/006d4016-cb97-4751-9c1b-83034e899d0e/volumes" Sep 13 20:11:34.428091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:34.427937 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faa57dad-5276-490f-8aa0-dea7ffe25b6c" path="/var/lib/kubelet/pods/faa57dad-5276-490f-8aa0-dea7ffe25b6c/volumes" Sep 13 20:11:38.088091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:38.088051 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:38.188370 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:38.188333 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:38.266826 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:38.266790 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:38.439590 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:38.439492 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:38.660620 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:38.660591 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:42.217600 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:42.217562 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:11:42.317661 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:11:42.317629 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:13.843245 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:13.843211 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:13.958141 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:13.958078 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:15.207316 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.207282 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:15.307848 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.307814 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:15.456539 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.456507 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:15.514596 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.514560 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm"] Sep 13 20:12:15.518167 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.518143 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.520409 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.520390 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 13 20:12:15.521271 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.521250 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ndpr4\"" Sep 13 20:12:15.521271 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.521258 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 13 20:12:15.526018 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.525993 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm"] Sep 13 20:12:15.585093 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.585065 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:15.608402 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.608366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a199750d-9863-4c73-ab73-3c532d1f043b-maas-api-tls\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.608402 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.608402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a199750d-9863-4c73-ab73-3c532d1f043b-metrics-certs\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.608596 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.608531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a199750d-9863-4c73-ab73-3c532d1f043b-openshift-service-ca\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.608596 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.608574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjp58\" (UniqueName: \"kubernetes.io/projected/a199750d-9863-4c73-ab73-3c532d1f043b-kube-api-access-rjp58\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.709379 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.709351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a199750d-9863-4c73-ab73-3c532d1f043b-maas-api-tls\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.709379 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.709380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a199750d-9863-4c73-ab73-3c532d1f043b-metrics-certs\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.709589 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.709434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a199750d-9863-4c73-ab73-3c532d1f043b-openshift-service-ca\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.709589 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.709460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjp58\" (UniqueName: \"kubernetes.io/projected/a199750d-9863-4c73-ab73-3c532d1f043b-kube-api-access-rjp58\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.710132 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.710110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a199750d-9863-4c73-ab73-3c532d1f043b-openshift-service-ca\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.711640 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.711619 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a199750d-9863-4c73-ab73-3c532d1f043b-maas-api-tls\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.711753 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.711677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a199750d-9863-4c73-ab73-3c532d1f043b-metrics-certs\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.717366 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.717344 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjp58\" (UniqueName: \"kubernetes.io/projected/a199750d-9863-4c73-ab73-3c532d1f043b-kube-api-access-rjp58\") pod \"maas-api-59f6566bbc-ps6rm\" (UID: \"a199750d-9863-4c73-ab73-3c532d1f043b\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.829853 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.829763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:15.868519 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.868486 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:15.969071 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.969036 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:15.974472 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.974443 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8"] Sep 13 20:12:15.978270 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.978248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:15.980645 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.980602 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 13 20:12:15.980645 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.980636 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-986gp\"" Sep 13 20:12:15.992381 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:15.992356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8"] Sep 13 20:12:16.107502 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.107473 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm"] Sep 13 20:12:16.110019 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:12:16.109991 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda199750d_9863_4c73_ab73_3c532d1f043b.slice/crio-e0ec0726c4b6dd69a2054059b78db9ad0417fe9e76190283c98516f0f203e3dc WatchSource:0}: Error finding container e0ec0726c4b6dd69a2054059b78db9ad0417fe9e76190283c98516f0f203e3dc: Status 404 returned error can't find the container with id e0ec0726c4b6dd69a2054059b78db9ad0417fe9e76190283c98516f0f203e3dc Sep 13 20:12:16.113091 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.113065 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5p7\" (UniqueName: \"kubernetes.io/projected/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-kube-api-access-6r5p7\") pod \"payload-pre-processing-59fbc47bf8-n7cr8\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:16.113234 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.113119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-n7cr8\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:16.214047 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.214013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-n7cr8\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:16.214565 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.214084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5p7\" (UniqueName: \"kubernetes.io/projected/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-kube-api-access-6r5p7\") pod \"payload-pre-processing-59fbc47bf8-n7cr8\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:16.214891 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.214868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-n7cr8\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:16.221135 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.221108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5p7\" (UniqueName: \"kubernetes.io/projected/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-kube-api-access-6r5p7\") pod \"payload-pre-processing-59fbc47bf8-n7cr8\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:16.289105 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.289063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:16.565688 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.565572 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8"] Sep 13 20:12:16.567976 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:12:16.567946 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dd9ebb_fcba_48b2_a1d9_f9ee874bc2ce.slice/crio-6bde3fc934552c6bc36d26d4aa77fe7ddb96369384fa7b1f2345734d7552bade WatchSource:0}: Error finding container 6bde3fc934552c6bc36d26d4aa77fe7ddb96369384fa7b1f2345734d7552bade: Status 404 returned error can't find the container with id 6bde3fc934552c6bc36d26d4aa77fe7ddb96369384fa7b1f2345734d7552bade Sep 13 20:12:16.931834 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.931706 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" event={"ID":"a199750d-9863-4c73-ab73-3c532d1f043b","Type":"ContainerStarted","Data":"e0ec0726c4b6dd69a2054059b78db9ad0417fe9e76190283c98516f0f203e3dc"} Sep 13 20:12:16.933216 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:16.933170 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" event={"ID":"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce","Type":"ContainerStarted","Data":"6bde3fc934552c6bc36d26d4aa77fe7ddb96369384fa7b1f2345734d7552bade"} Sep 13 20:12:18.946559 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:18.946477 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" event={"ID":"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce","Type":"ContainerStarted","Data":"507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb"} Sep 13 20:12:18.946993 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:18.946650 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:18.948046 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:18.948022 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" event={"ID":"a199750d-9863-4c73-ab73-3c532d1f043b","Type":"ContainerStarted","Data":"38fdce7a0c89cc43b612c04a7705b5f040d21273475e9a263b1ede2a11a7992d"} Sep 13 20:12:18.948160 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:18.948077 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:18.962786 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:18.962735 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" podStartSLOduration=1.8640569999999999 podStartE2EDuration="3.962720449s" totalImagesPullingTime="2.098663449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:12:18.960611746 +0000 UTC m=+835.043756671" watchObservedRunningTime="2026-09-13 20:12:18.962720449 +0000 UTC m=+835.045865380" Sep 13 20:12:18.975554 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:18.975505 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" podStartSLOduration=1.416925536 podStartE2EDuration="3.975490431s" totalImagesPullingTime="2.558564895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:12:18.974497631 +0000 UTC m=+835.057642569" watchObservedRunningTime="2026-09-13 20:12:18.975490431 +0000 UTC m=+835.058635345" Sep 13 20:12:24.958733 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:24.958698 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-ps6rm" Sep 13 20:12:25.932350 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:25.932320 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:26.028721 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:26.028689 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:29.310302 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.310265 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k"] Sep 13 20:12:29.314589 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.314564 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.316819 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.316796 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 13 20:12:29.317080 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.317059 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 13 20:12:29.317746 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.317726 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 13 20:12:29.317842 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.317772 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-khw6f\"" Sep 13 20:12:29.322829 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.322808 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k"] Sep 13 20:12:29.347265 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.347239 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:12:29.431332 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.431296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.431332 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.431331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf6629-7de4-4802-aafe-50dfed77c7c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.431545 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.431351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkthv\" (UniqueName: \"kubernetes.io/projected/2bcf6629-7de4-4802-aafe-50dfed77c7c9-kube-api-access-wkthv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.431545 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.431465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.431545 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.431516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.431545 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.431544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532328 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532510 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532510 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532631 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532631 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf6629-7de4-4802-aafe-50dfed77c7c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532631 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkthv\" (UniqueName: \"kubernetes.io/projected/2bcf6629-7de4-4802-aafe-50dfed77c7c9-kube-api-access-wkthv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532791 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532747 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532845 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532786 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.532894 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.532844 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.534811 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.534785 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcf6629-7de4-4802-aafe-50dfed77c7c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.535220 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.535178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcf6629-7de4-4802-aafe-50dfed77c7c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.541735 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.541708 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkthv\" (UniqueName: \"kubernetes.io/projected/2bcf6629-7de4-4802-aafe-50dfed77c7c9-kube-api-access-wkthv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k\" (UID: \"2bcf6629-7de4-4802-aafe-50dfed77c7c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.626340 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.626256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:29.902321 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.902294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k"] Sep 13 20:12:29.904456 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:12:29.904428 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcf6629_7de4_4802_aafe_50dfed77c7c9.slice/crio-f5a4ea75b3c3a95674c1a256fe9cb1f8bc222c202c33b28ad90ac30df106f85b WatchSource:0}: Error finding container f5a4ea75b3c3a95674c1a256fe9cb1f8bc222c202c33b28ad90ac30df106f85b: Status 404 returned error can't find the container with id f5a4ea75b3c3a95674c1a256fe9cb1f8bc222c202c33b28ad90ac30df106f85b Sep 13 20:12:29.956481 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:29.956447 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:12:30.012603 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:30.012570 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" event={"ID":"2bcf6629-7de4-4802-aafe-50dfed77c7c9","Type":"ContainerStarted","Data":"f5a4ea75b3c3a95674c1a256fe9cb1f8bc222c202c33b28ad90ac30df106f85b"} Sep 13 20:12:32.024692 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:32.024657 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" event={"ID":"2bcf6629-7de4-4802-aafe-50dfed77c7c9","Type":"ContainerStarted","Data":"d650a3f58355d753df6ee2d4eb88b18c4442ff8b5e79a133f63e6d5d30a21407"} Sep 13 20:12:32.040328 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:32.040281 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" podStartSLOduration=1.9294299160000001 podStartE2EDuration="3.040264886s" totalImagesPullingTime="1.11083497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:12:32.039167689 +0000 UTC m=+848.122312636" watchObservedRunningTime="2026-09-13 20:12:32.040264886 +0000 UTC m=+848.123409802" Sep 13 20:12:39.626593 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:39.626550 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:39.627059 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:39.626641 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:39.639179 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:39.639150 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:12:40.078960 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:12:40.078925 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-dc22k" Sep 13 20:13:03.826692 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:03.826661 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:13:18.738978 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:18.738896 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:13:37.434255 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:37.434224 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:13:42.106161 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.106121 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b"] Sep 13 20:13:42.109562 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.109545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.111538 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.111512 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 13 20:13:42.118209 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.118157 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b"] Sep 13 20:13:42.182610 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.182570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.182610 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.182611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.182856 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.182631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lktc\" (UniqueName: \"kubernetes.io/projected/52c19038-40b0-42cd-a4b5-195365470105-kube-api-access-6lktc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.182856 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.182703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.182856 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.182749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c19038-40b0-42cd-a4b5-195365470105-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.182856 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.182840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.208419 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.208388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:13:42.283947 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.283908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.283947 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.283953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.284243 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.283972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lktc\" (UniqueName: \"kubernetes.io/projected/52c19038-40b0-42cd-a4b5-195365470105-kube-api-access-6lktc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.284243 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.283996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.284243 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.284034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c19038-40b0-42cd-a4b5-195365470105-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.284243 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.284066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.284473 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.284416 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.284531 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.284475 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.284531 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.284506 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.286428 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.286397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52c19038-40b0-42cd-a4b5-195365470105-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.286642 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.286621 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52c19038-40b0-42cd-a4b5-195365470105-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.291313 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.291292 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lktc\" (UniqueName: \"kubernetes.io/projected/52c19038-40b0-42cd-a4b5-195365470105-kube-api-access-6lktc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b\" (UID: \"52c19038-40b0-42cd-a4b5-195365470105\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.421774 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.421687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:42.693972 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:42.693898 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b"] Sep 13 20:13:42.697296 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:13:42.697264 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c19038_40b0_42cd_a4b5_195365470105.slice/crio-c498ed36430909577380d3558ed169018c261ea4890138da99b1893684406ecd WatchSource:0}: Error finding container c498ed36430909577380d3558ed169018c261ea4890138da99b1893684406ecd: Status 404 returned error can't find the container with id c498ed36430909577380d3558ed169018c261ea4890138da99b1893684406ecd Sep 13 20:13:43.382941 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:43.382901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" event={"ID":"52c19038-40b0-42cd-a4b5-195365470105","Type":"ContainerStarted","Data":"51f45fc7bd3f7c4101fe865aa37914db65fd3e5555cda7bef5ea6bf6bb07d160"} Sep 13 20:13:43.383418 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:43.382956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" event={"ID":"52c19038-40b0-42cd-a4b5-195365470105","Type":"ContainerStarted","Data":"c498ed36430909577380d3558ed169018c261ea4890138da99b1893684406ecd"} Sep 13 20:13:43.398999 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:43.398948 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" podStartSLOduration=1.221374313 podStartE2EDuration="1.398935082s" totalImagesPullingTime="177.560769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:13:43.397037947 +0000 UTC m=+919.480182882" watchObservedRunningTime="2026-09-13 20:13:43.398935082 +0000 UTC m=+919.482079996" Sep 13 20:13:50.708320 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.708283 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn"] Sep 13 20:13:50.712261 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.712230 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.714522 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.714495 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 13 20:13:50.720902 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.720876 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn"] Sep 13 20:13:50.732460 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.732429 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:13:50.866530 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.866483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.866707 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.866558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.866707 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.866584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.866707 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.866618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c1767c-5028-4e58-868d-4a7c64611cf1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.866707 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.866674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.866849 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.866723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxlb\" (UniqueName: \"kubernetes.io/projected/b8c1767c-5028-4e58-868d-4a7c64611cf1-kube-api-access-cnxlb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.967818 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.967727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.967818 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.967781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.967818 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.967800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.968116 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.967832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c1767c-5028-4e58-868d-4a7c64611cf1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.968116 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.967857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.968116 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.967895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxlb\" (UniqueName: \"kubernetes.io/projected/b8c1767c-5028-4e58-868d-4a7c64611cf1-kube-api-access-cnxlb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.968315 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.968288 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.968480 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.968447 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.968613 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.968459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.970323 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.970298 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c1767c-5028-4e58-868d-4a7c64611cf1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.970490 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.970471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c1767c-5028-4e58-868d-4a7c64611cf1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:50.974868 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:50.974843 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxlb\" (UniqueName: \"kubernetes.io/projected/b8c1767c-5028-4e58-868d-4a7c64611cf1-kube-api-access-cnxlb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jc6rn\" (UID: \"b8c1767c-5028-4e58-868d-4a7c64611cf1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:51.025673 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:51.025627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:13:51.299628 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:51.299599 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn"] Sep 13 20:13:51.301504 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:13:51.301473 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c1767c_5028_4e58_868d_4a7c64611cf1.slice/crio-4549ce6c0522ce1a07e9248b42131df99f6eeb0c9587a45d5478adf211f67410 WatchSource:0}: Error finding container 4549ce6c0522ce1a07e9248b42131df99f6eeb0c9587a45d5478adf211f67410: Status 404 returned error can't find the container with id 4549ce6c0522ce1a07e9248b42131df99f6eeb0c9587a45d5478adf211f67410 Sep 13 20:13:51.302879 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:51.302858 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 13 20:13:51.425953 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:51.425918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" event={"ID":"b8c1767c-5028-4e58-868d-4a7c64611cf1","Type":"ContainerStarted","Data":"4549ce6c0522ce1a07e9248b42131df99f6eeb0c9587a45d5478adf211f67410"} Sep 13 20:13:52.422041 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:52.422009 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:52.422468 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:52.422077 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:52.437595 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:52.437558 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:52.438993 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:52.438963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" event={"ID":"b8c1767c-5028-4e58-868d-4a7c64611cf1","Type":"ContainerStarted","Data":"6a6ebbef8183feb1ee313e4fd62a552af60ee821b44edb5b250039de705d3c3c"} Sep 13 20:13:52.451515 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:52.451488 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4dz4b" Sep 13 20:13:52.481065 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:13:52.480957 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" podStartSLOduration=2.30654809 podStartE2EDuration="2.480937661s" totalImagesPullingTime="174.389571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-13 20:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:13:52.480371764 +0000 UTC m=+928.563516678" watchObservedRunningTime="2026-09-13 20:13:52.480937661 +0000 UTC m=+928.564082575" Sep 13 20:14:01.025901 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:14:01.025860 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:14:01.026386 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:14:01.025922 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:14:01.038370 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:14:01.038342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:14:01.498250 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:14:01.498156 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jc6rn" Sep 13 20:14:05.732241 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:14:05.732180 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:15:12.805833 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:12.805799 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:15:14.161612 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.161579 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:15:14.258050 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.258015 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8"] Sep 13 20:15:14.258349 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.258317 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" podUID="76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" containerName="payload-pre-processing" containerID="cri-o://507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb" gracePeriod=30 Sep 13 20:15:14.279911 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.279885 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:15:14.517040 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.517015 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:15:14.614433 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.614393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r5p7\" (UniqueName: \"kubernetes.io/projected/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-kube-api-access-6r5p7\") pod \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " Sep 13 20:15:14.614617 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.614503 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-plugins-config-volume\") pod \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\" (UID: \"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce\") " Sep 13 20:15:14.614902 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.614868 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" (UID: "76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 13 20:15:14.616680 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.616647 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-kube-api-access-6r5p7" (OuterVolumeSpecName: "kube-api-access-6r5p7") pod "76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" (UID: "76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce"). InnerVolumeSpecName "kube-api-access-6r5p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 13 20:15:14.716044 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.715959 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-plugins-config-volume\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:15:14.716044 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.715989 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5p7\" (UniqueName: \"kubernetes.io/projected/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce-kube-api-access-6r5p7\") on node \"ip-10-0-141-146.ec2.internal\" DevicePath \"\"" Sep 13 20:15:14.862331 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.862291 2578 generic.go:356] "Generic (PLEG): container finished" podID="76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" containerID="507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb" exitCode=0 Sep 13 20:15:14.862523 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.862335 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" Sep 13 20:15:14.862523 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.862375 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" event={"ID":"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce","Type":"ContainerDied","Data":"507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb"} Sep 13 20:15:14.862523 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.862432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8" event={"ID":"76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce","Type":"ContainerDied","Data":"6bde3fc934552c6bc36d26d4aa77fe7ddb96369384fa7b1f2345734d7552bade"} Sep 13 20:15:14.862523 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.862455 2578 scope.go:117] "RemoveContainer" containerID="507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb" Sep 13 20:15:14.873920 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.873899 2578 scope.go:117] "RemoveContainer" containerID="507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb" Sep 13 20:15:14.874350 ip-10-0-141-146 kubenswrapper[2578]: E0913 20:15:14.874321 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb\": container with ID starting with 507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb not found: ID does not exist" containerID="507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb" Sep 13 20:15:14.874452 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.874360 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb"} err="failed to get container status \"507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb\": rpc error: code = NotFound desc = could not find container \"507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb\": container with ID starting with 507b1dbcd35d37deb4a3f15350c4f50bd47dd2ed93b82e986e85e1a69a7d6deb not found: ID does not exist" Sep 13 20:15:14.885644 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.885623 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8"] Sep 13 20:15:14.895610 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:14.895589 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-n7cr8"] Sep 13 20:15:16.421038 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:16.421002 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" path="/var/lib/kubelet/pods/76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce/volumes" Sep 13 20:15:25.998211 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:25.998150 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:15:56.858977 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:15:56.858916 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:17:50.161920 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:17:50.161819 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:18:54.748520 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:18:54.748483 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:18:55.905487 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:18:55.905451 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:18:57.498273 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:18:57.498219 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8lrwz_7421f4bb-2546-4d47-9444-86fb057812c3/global-pull-secret-syncer/0.log" Sep 13 20:18:57.633789 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:18:57.633751 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" Sep 13 20:18:57.727383 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:18:57.727348 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-146.ec2.internal_c92652107ed38ad17b29af94e128de12/haproxy/0.log" Sep 13 20:19:01.480702 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:01.480670 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:19:01.578201 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:01.578154 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:19:02.059581 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:02.059551 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-cgd8h_d4a93942-6044-4477-a5ce-86ad2c74562c/manager/0.log" Sep 13 20:19:02.081322 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:02.081290 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-jtlmn_b8a0df6f-6057-452f-aa5a-d0453ca6621b/manager/0.log" Sep 13 20:19:02.114337 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:02.114309 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9h2f9_14afd400-a7ad-4787-8f96-1bde0831d319/kuadrant-console-plugin/0.log" Sep 13 20:19:02.173393 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:02.173358 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hhvtg_8cbf8a7d-12f1-4f77-9081-5897b71a62eb/manager/0.log" Sep 13 20:19:02.238992 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:02.238966 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-97jst_fce9dc1c-61dd-4488-b0da-d2b636852f38/manager/0.log" Sep 13 20:19:04.014537 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.014501 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-krnsb_5a4f4bcd-70c7-4e0d-842d-be06c2aea763/kube-state-metrics/0.log" Sep 13 20:19:04.029112 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.029090 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-krnsb_5a4f4bcd-70c7-4e0d-842d-be06c2aea763/kube-rbac-proxy-main/0.log" Sep 13 20:19:04.047811 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.047785 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-krnsb_5a4f4bcd-70c7-4e0d-842d-be06c2aea763/kube-rbac-proxy-self/0.log" Sep 13 20:19:04.109052 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.109022 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-btvc2_e38f87e0-48bf-427b-85da-c36d319cccb8/monitoring-plugin/0.log" Sep 13 20:19:04.299932 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.299833 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvj6j_f563bbe7-aa7b-4a12-967a-e01253aa7d72/node-exporter/0.log" Sep 13 20:19:04.316892 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.316863 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvj6j_f563bbe7-aa7b-4a12-967a-e01253aa7d72/kube-rbac-proxy/0.log" Sep 13 20:19:04.336776 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.336754 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvj6j_f563bbe7-aa7b-4a12-967a-e01253aa7d72/init-textfile/0.log" Sep 13 20:19:04.781745 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.781709 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7484c56547-qzlfj_fd3fc4b5-3397-4372-bc27-57dcf2f23f94/thanos-query/0.log" Sep 13 20:19:04.824552 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.824522 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7484c56547-qzlfj_fd3fc4b5-3397-4372-bc27-57dcf2f23f94/kube-rbac-proxy-web/0.log" Sep 13 20:19:04.846062 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.846025 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7484c56547-qzlfj_fd3fc4b5-3397-4372-bc27-57dcf2f23f94/kube-rbac-proxy/0.log" Sep 13 20:19:04.865612 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.865583 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7484c56547-qzlfj_fd3fc4b5-3397-4372-bc27-57dcf2f23f94/prom-label-proxy/0.log" Sep 13 20:19:04.887521 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.887445 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7484c56547-qzlfj_fd3fc4b5-3397-4372-bc27-57dcf2f23f94/kube-rbac-proxy-rules/0.log" Sep 13 20:19:04.905314 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:04.905287 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7484c56547-qzlfj_fd3fc4b5-3397-4372-bc27-57dcf2f23f94/kube-rbac-proxy-metrics/0.log" Sep 13 20:19:07.556925 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:07.556888 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-plznc_9ee869a4-5438-45db-b9e4-ee6dfd603156/download-server/0.log" Sep 13 20:19:08.109024 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.108987 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg"] Sep 13 20:19:08.109462 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.109434 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" containerName="payload-pre-processing" Sep 13 20:19:08.109462 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.109458 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" containerName="payload-pre-processing" Sep 13 20:19:08.109630 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.109576 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76dd9ebb-fcba-48b2-a1d9-f9ee874bc2ce" containerName="payload-pre-processing" Sep 13 20:19:08.112701 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.112685 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.114725 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.114708 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwr95\"/\"openshift-service-ca.crt\"" Sep 13 20:19:08.114843 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.114823 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mwr95\"/\"kube-root-ca.crt\"" Sep 13 20:19:08.115764 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.115744 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mwr95\"/\"default-dockercfg-kqvcq\"" Sep 13 20:19:08.121333 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.121310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg"] Sep 13 20:19:08.238515 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.238475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-proc\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.238715 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.238522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdfl\" (UniqueName: \"kubernetes.io/projected/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-kube-api-access-sxdfl\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.238715 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.238622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-podres\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.238827 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.238717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-sys\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.238827 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.238760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-lib-modules\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.260879 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.260844 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-ps4lh_f9228fd1-3e06-48cc-9d0c-8168c74ae4ea/volume-data-source-validator/0.log" Sep 13 20:19:08.339493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-sys\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339493 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-lib-modules\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339714 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-proc\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339714 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339584 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-proc\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339714 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339584 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-sys\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339714 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdfl\" (UniqueName: \"kubernetes.io/projected/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-kube-api-access-sxdfl\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339714 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-podres\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339714 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-lib-modules\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.339942 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.339791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-podres\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.347382 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.347353 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdfl\" (UniqueName: \"kubernetes.io/projected/2f3ceb1b-627b-49c7-a8e2-36934801b9ee-kube-api-access-sxdfl\") pod \"perf-node-gather-daemonset-nn2fg\" (UID: \"2f3ceb1b-627b-49c7-a8e2-36934801b9ee\") " pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.422836 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.422760 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:08.691980 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:08.691954 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg"] Sep 13 20:19:08.694833 ip-10-0-141-146 kubenswrapper[2578]: W0913 20:19:08.694793 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2f3ceb1b_627b_49c7_a8e2_36934801b9ee.slice/crio-db8d4aae5d9857f60ed660f21798bc380efbac94ba950be2e89ff081a66be48d WatchSource:0}: Error finding container db8d4aae5d9857f60ed660f21798bc380efbac94ba950be2e89ff081a66be48d: Status 404 returned error can't find the container with id db8d4aae5d9857f60ed660f21798bc380efbac94ba950be2e89ff081a66be48d Sep 13 20:19:09.028350 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.028315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" event={"ID":"2f3ceb1b-627b-49c7-a8e2-36934801b9ee","Type":"ContainerStarted","Data":"7e74390272a23b236f1f36aa3a0ee2b065317a45b524599b290aa7ad20736017"} Sep 13 20:19:09.028526 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.028369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" event={"ID":"2f3ceb1b-627b-49c7-a8e2-36934801b9ee","Type":"ContainerStarted","Data":"db8d4aae5d9857f60ed660f21798bc380efbac94ba950be2e89ff081a66be48d"} Sep 13 20:19:09.028526 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.028391 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:09.042378 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.042326 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" podStartSLOduration=1.042308627 podStartE2EDuration="1.042308627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-13 20:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-13 20:19:09.041667683 +0000 UTC m=+1245.124812600" watchObservedRunningTime="2026-09-13 20:19:09.042308627 +0000 UTC m=+1245.125453541" Sep 13 20:19:09.372578 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.372485 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jcz7f_fd881c97-f6a3-46b3-922c-451a18fdfafd/dns/0.log" Sep 13 20:19:09.391553 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.391529 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jcz7f_fd881c97-f6a3-46b3-922c-451a18fdfafd/kube-rbac-proxy/0.log" Sep 13 20:19:09.458556 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.458524 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x6bb4_f676cc21-4b60-4544-b5b8-1c7d297575c4/dns-node-resolver/0.log" Sep 13 20:19:09.967735 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:09.967329 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54ccd6f8b8-b5z66_cd215c20-90f9-4732-a877-544e958b54be/registry/0.log" Sep 13 20:19:10.031071 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:10.031035 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xfg7g_9746c0ef-e12e-4b7e-9b92-e3878cc381d9/node-ca/0.log" Sep 13 20:19:11.002067 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:11.002025 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4d676b-vvxlj_3fe165cf-58f8-4dea-82e3-d6e1d48ae04a/kube-auth-proxy/0.log" Sep 13 20:19:11.726293 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:11.726263 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-89mtv_88db34e6-cfe4-4767-9054-ba703ed75960/serve-healthcheck-canary/0.log" Sep 13 20:19:12.215880 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:12.215845 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-shgsc_1f476a24-96a1-4723-ba1d-45bf3df32556/insights-operator/0.log" Sep 13 20:19:12.216311 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:12.215935 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-shgsc_1f476a24-96a1-4723-ba1d-45bf3df32556/insights-operator/1.log" Sep 13 20:19:12.296974 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:12.296945 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl98d_82635252-2416-496f-b0ee-50725e625c15/kube-rbac-proxy/0.log" Sep 13 20:19:12.317404 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:12.317371 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl98d_82635252-2416-496f-b0ee-50725e625c15/exporter/0.log" Sep 13 20:19:12.340070 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:12.340043 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl98d_82635252-2416-496f-b0ee-50725e625c15/extractor/0.log" Sep 13 20:19:14.421768 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:14.421731 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-97lfk_2ea06763-470f-4bb3-b103-0165a34d72ba/manager/0.log" Sep 13 20:19:14.465199 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:14.465163 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rb8rs_a7571ab8-b66f-4aa5-b026-def4f907dd22/manager/0.log" Sep 13 20:19:14.595496 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:14.595460 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79dc4cfddb-phnhf_b4288a23-4050-49a9-87b8-490cf144ff50/manager/0.log" Sep 13 20:19:14.613328 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:14.613301 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79dc4cfddb-phnhf_b4288a23-4050-49a9-87b8-490cf144ff50/copy-manifests/0.log" Sep 13 20:19:15.043848 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:15.043820 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mwr95/perf-node-gather-daemonset-nn2fg" Sep 13 20:19:15.676297 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:15.676261 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:19:15.734855 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:15.734824 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7hslg_343b6278-de15-4a40-9166-8a253a28bbf6/konnectivity-agent/0.log" limit=1024 Sep 13 20:19:15.886196 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:15.886152 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kgzrk_307e3d7e-7cd6-49a7-9845-79a1aa12ed22/openshift-lws-operator/0.log" Sep 13 20:19:17.818216 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:17.818160 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4d676b-vvxlj_3fe165cf-58f8-4dea-82e3-d6e1d48ae04a/kube-auth-proxy/0.log" Sep 13 20:19:19.139451 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:19.139410 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d4d676b-vvxlj_3fe165cf-58f8-4dea-82e3-d6e1d48ae04a/kube-auth-proxy/0.log" Sep 13 20:19:21.125113 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:21.125081 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-q2svn_b6fd1d72-04ee-4147-a416-571e087fe5ca/migrator/0.log" Sep 13 20:19:21.143140 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:21.143114 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-q2svn_b6fd1d72-04ee-4147-a416-571e087fe5ca/graceful-termination/0.log" Sep 13 20:19:22.500816 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.500780 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7xzwk_b99c914e-69ca-4aed-9855-b34b7ef4b856/kube-multus/0.log" Sep 13 20:19:22.656443 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.656403 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phbjh_3898959b-2d2f-4923-9900-288c55d80efc/kube-multus-additional-cni-plugins/0.log" Sep 13 20:19:22.676014 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.675991 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phbjh_3898959b-2d2f-4923-9900-288c55d80efc/egress-router-binary-copy/0.log" Sep 13 20:19:22.694448 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.694426 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phbjh_3898959b-2d2f-4923-9900-288c55d80efc/cni-plugins/0.log" Sep 13 20:19:22.712924 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.712900 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phbjh_3898959b-2d2f-4923-9900-288c55d80efc/bond-cni-plugin/0.log" Sep 13 20:19:22.733249 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.733225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phbjh_3898959b-2d2f-4923-9900-288c55d80efc/routeoverride-cni/0.log" Sep 13 20:19:22.752250 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.752107 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phbjh_3898959b-2d2f-4923-9900-288c55d80efc/whereabouts-cni-bincopy/0.log" Sep 13 20:19:22.782938 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:22.782912 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phbjh_3898959b-2d2f-4923-9900-288c55d80efc/whereabouts-cni/0.log" Sep 13 20:19:23.090196 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:23.090153 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xk2rl_2473e3dd-c0cf-4159-8b9d-230c808ea682/network-metrics-daemon/0.log" Sep 13 20:19:23.106897 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:23.106870 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xk2rl_2473e3dd-c0cf-4159-8b9d-230c808ea682/kube-rbac-proxy/0.log" Sep 13 20:19:24.279580 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.279550 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/ovn-controller/0.log" Sep 13 20:19:24.300372 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.300339 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/ovn-acl-logging/0.log" Sep 13 20:19:24.326221 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.326172 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/kube-rbac-proxy-node/0.log" Sep 13 20:19:24.348700 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.348654 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/kube-rbac-proxy-ovn-metrics/0.log" Sep 13 20:19:24.364345 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.364317 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/northd/0.log" Sep 13 20:19:24.382142 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.382102 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/nbdb/0.log" Sep 13 20:19:24.400446 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.400425 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/sbdb/0.log" Sep 13 20:19:24.454150 ip-10-0-141-146 kubenswrapper[2578]: I0913 20:19:24.454070 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sq2fp_1cf36c1b-4c27-46c0-acff-e80da5efc3b0/ovnkube-controller/0.log"