Jul 15 13:29:18.369648 ip-10-0-129-47 systemd[1]: Starting Kubernetes Kubelet... Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.741991 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748085 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748102 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748105 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748109 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748112 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748115 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748117 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:29:18.797217 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748120 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748123 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748127 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748130 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748132 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748135 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748137 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748140 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748143 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748146 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748149 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748154 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748157 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748159 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748161 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748164 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748166 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748169 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748172 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:29:18.798770 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748174 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748177 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748179 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748182 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748184 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748187 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748189 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748192 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748196 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748200 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748203 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748205 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748208 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748210 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748213 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748216 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748219 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748222 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748225 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:29:18.799273 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748227 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748230 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748234 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748236 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748239 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748242 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748246 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748248 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748251 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748253 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748256 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748258 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748263 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748267 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748269 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748272 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748275 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748277 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748280 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748282 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:29:18.799819 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748285 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748287 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748290 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748292 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748295 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748298 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748301 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748303 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748306 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748308 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748311 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748313 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748316 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748318 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748321 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748324 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748329 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748333 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748342 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748345 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:29:18.800357 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748348 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748351 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748354 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748358 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748361 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748363 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748366 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748369 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748372 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748374 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748377 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748380 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748383 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.748386 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749159 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749168 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749171 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749175 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749178 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749181 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:29:18.800906 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749183 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749186 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749190 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749193 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749195 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749198 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749200 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749203 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749205 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749208 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749210 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749213 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749216 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749218 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749222 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749224 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749227 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749230 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749232 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749234 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:29:18.801464 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749237 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749240 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749243 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749247 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749250 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749253 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749256 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749259 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749263 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749267 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749269 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749272 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749275 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749277 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749280 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749282 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749285 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749287 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749290 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:29:18.801980 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749292 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749295 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749297 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749300 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749302 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749306 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749308 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749310 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749313 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749316 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749318 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749320 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749323 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749325 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749328 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749332 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749336 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749339 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749344 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:29:18.802630 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749348 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749351 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749354 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749356 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749359 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749361 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749364 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749366 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749369 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749371 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749374 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749376 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749379 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749381 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749383 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749386 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749389 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749391 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749408 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749413 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:29:18.803122 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749415 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749418 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749420 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749423 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749425 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749428 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749430 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749434 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749436 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749441 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749445 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749450 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749453 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749456 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749459 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749835 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749839 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749843 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749845 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749848 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:29:18.803696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749850 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749853 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749855 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749858 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749861 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749863 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749866 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749868 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749871 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749873 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749876 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749878 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749881 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749884 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749887 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749889 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749892 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749895 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749897 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749899 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:29:18.804206 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749902 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749904 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749906 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749911 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749914 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749917 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749921 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749925 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749928 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749930 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749933 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749936 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749938 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749941 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749943 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749945 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749948 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749950 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749953 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749955 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:29:18.895119 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749958 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:29:18.817108 ip-10-0-129-47 systemd[1]: Started Kubernetes Kubelet. Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749960 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749963 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749965 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749968 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749970 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749973 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749975 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749978 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749980 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749982 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749985 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749987 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749991 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749995 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.749999 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750002 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750005 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750008 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:29:18.895863 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750012 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750015 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750017 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750020 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750022 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750025 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750027 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750030 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750032 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750035 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750037 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750043 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750046 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750048 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750051 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750053 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750055 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750058 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750061 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750063 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:29:18.900292 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750066 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750068 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750071 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750073 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750075 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750078 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750080 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750082 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750086 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750089 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750091 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750094 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750097 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750100 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750102 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750178 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750186 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750191 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750195 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750199 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 13:29:18.900986 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750202 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750206 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750210 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750213 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750216 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750219 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750223 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750226 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750229 2576 flags.go:64] FLAG: --cgroup-root="" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750231 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750234 2576 flags.go:64] FLAG: --client-ca-file="" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750238 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750241 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750245 2576 flags.go:64] FLAG: --cluster-domain="" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750248 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750251 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750254 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750257 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750261 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750263 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750268 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750271 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750274 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750277 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 13:29:18.901621 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750281 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750284 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750287 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750291 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750294 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750298 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750300 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750313 2576 flags.go:64] FLAG: --enable-server="true" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750316 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750321 2576 flags.go:64] FLAG: --event-burst="100" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750324 2576 flags.go:64] FLAG: --event-qps="50" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750327 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750330 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750332 2576 flags.go:64] FLAG: --eviction-hard="" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750336 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750339 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750342 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750345 2576 flags.go:64] FLAG: --eviction-soft="" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750348 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750350 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750354 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750357 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750360 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750362 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750365 2576 flags.go:64] FLAG: --feature-gates="" Jul 15 13:29:18.902255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750369 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750371 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750374 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750379 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750382 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750384 2576 flags.go:64] FLAG: --help="false" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750387 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-129-47.ec2.internal" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750392 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750405 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750409 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750412 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750415 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750418 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750421 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750424 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750427 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750430 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750433 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750436 2576 flags.go:64] FLAG: --kube-reserved="" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750438 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750441 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750444 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750447 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750449 2576 flags.go:64] FLAG: --lock-file="" Jul 15 13:29:18.903106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750452 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750455 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750458 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750464 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750467 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750470 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750473 2576 flags.go:64] FLAG: --logging-format="text" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750475 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750478 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750481 2576 flags.go:64] FLAG: --manifest-url="" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750484 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750490 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750493 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750497 2576 flags.go:64] FLAG: --max-pods="110" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750500 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750504 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750507 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750510 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750513 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750516 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750518 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750526 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750529 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750532 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 13:29:18.903793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750535 2576 flags.go:64] FLAG: --pod-cidr="" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750538 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750543 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750546 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750549 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750551 2576 flags.go:64] FLAG: --port="10250" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750554 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750557 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d9d8c90ba31bc3e3" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750560 2576 flags.go:64] FLAG: --qos-reserved="" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750563 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750565 2576 flags.go:64] FLAG: --register-node="true" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750568 2576 flags.go:64] FLAG: --register-with-taints="" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750571 2576 flags.go:64] FLAG: --registry-burst="10" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750575 2576 flags.go:64] FLAG: --registry-qps="5" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750578 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750580 2576 flags.go:64] FLAG: --reserved-memory="" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750584 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750587 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750589 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750592 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750596 2576 flags.go:64] FLAG: --runonce="false" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750599 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750602 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750607 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750609 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 13:29:18.904493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750612 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750615 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750617 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750620 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750623 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750626 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750628 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750631 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750634 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750637 2576 flags.go:64] FLAG: --system-cgroups="" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750640 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750645 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750647 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750650 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750654 2576 flags.go:64] FLAG: --tls-min-version="" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750657 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750660 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750662 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750665 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750668 2576 flags.go:64] FLAG: --v="2" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750672 2576 flags.go:64] FLAG: --version="false" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750681 2576 flags.go:64] FLAG: --vmodule="" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750685 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.750688 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750802 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:29:18.905576 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750806 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750809 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750813 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750816 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750818 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750822 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750825 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750829 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750833 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750835 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750838 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750841 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750844 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750847 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750850 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750852 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750856 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750860 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750863 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:29:18.906421 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750865 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750867 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750870 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750872 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750875 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750877 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750880 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750882 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750885 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750887 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750890 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750893 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750895 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750899 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750903 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750906 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750909 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750911 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750915 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750918 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:29:19.057296 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750921 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750923 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750926 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750928 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750931 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750933 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750936 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750938 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750941 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750943 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750946 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750948 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750951 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750953 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750955 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750958 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750960 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750963 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750965 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750967 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:29:19.058184 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750970 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750972 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750974 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750977 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750980 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750982 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750985 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750989 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750991 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750994 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.750998 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751000 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751003 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751005 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751008 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751010 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751013 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751016 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751018 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751020 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:29:19.059096 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751023 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751025 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751028 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751030 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751032 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751035 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751038 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751040 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751043 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751045 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751047 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751050 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751052 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751055 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751057 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751059 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751062 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751065 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:29:19.060169 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.751067 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.751640 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.758357 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.758374 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.759123 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.759131 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759211 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759216 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759219 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759223 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759226 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759229 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759231 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759234 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:29:19.061065 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759237 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759240 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759242 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759245 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759248 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759251 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759253 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759256 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759259 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759261 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759264 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759267 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759270 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759273 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759275 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759278 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759281 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759283 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759286 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759288 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:29:19.061728 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759291 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759294 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759297 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759300 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759302 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759305 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759307 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759310 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759312 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759315 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759318 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759323 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759326 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759328 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759331 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759334 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759336 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759340 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759342 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759345 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:29:19.062589 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759347 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759350 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759352 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759355 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759357 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759360 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759363 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759365 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759368 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759377 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759380 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759382 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759385 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759387 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759390 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759392 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759412 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759416 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759420 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:29:19.063662 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759424 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759428 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759430 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759434 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759439 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759441 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759444 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759446 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759449 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759452 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759454 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759456 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759460 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759464 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759467 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759470 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759472 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759475 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759477 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:29:19.064738 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759480 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759482 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759485 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759487 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759496 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759499 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759501 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759504 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759506 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759508 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759511 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759513 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759516 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.759521 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759686 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:29:19.065603 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759691 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759695 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759700 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759703 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759706 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759709 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759712 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759715 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759718 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759721 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759723 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759726 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759728 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759731 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759733 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759736 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759738 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759741 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759743 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759746 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:29:19.066257 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759748 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759751 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759759 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759762 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759764 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759767 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759769 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759773 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759776 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759779 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759781 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759784 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759787 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759789 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759793 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759795 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759797 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759800 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759802 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759806 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:29:19.067128 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759809 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759812 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759815 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759817 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759820 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759823 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759825 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759828 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759830 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759833 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759835 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759838 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759840 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759842 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759845 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759853 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759856 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759858 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759860 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759864 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:29:19.067982 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759867 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759870 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759872 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759875 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759877 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759879 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759883 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759893 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759896 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759898 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759901 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759903 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759906 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759908 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759910 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759913 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759915 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759919 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759922 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:29:19.068870 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759925 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759927 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759930 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759933 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759935 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759937 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759940 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759942 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759945 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759954 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759956 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759959 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759962 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759966 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759969 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759972 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759974 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759977 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:29:19.069758 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:18.759979 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.759985 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.760098 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.762191 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.763278 2576 server.go:1019] "Starting client certificate rotation" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.763384 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.763436 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.765676 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.769889 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.777537 2576 log.go:25] "Validated CRI v1 runtime API" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.780053 2576 log.go:25] "Validated CRI v1 image API" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.780925 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 13:29:19.070660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.784413 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c12b609e-e729-45b4-a03c-dacde5466278:/dev/nvme0n1p4 f6e3d487-6991-4e48-ae2a-a82cd5bd2f1e:/dev/nvme0n1p3] Jul 15 13:29:19.071302 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.784430 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 13:29:19.071302 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.792229 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.795559 2576 manager.go:216] Machine: {Timestamp:2026-07-15 13:29:18.793096962 +0000 UTC m=+0.310721151 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201378 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec255bbdde05e6ca1a9bec43112551b0 SystemUUID:ec255bbd-de05-e6ca-1a9b-ec43112551b0 BootID:a8e7f79c-0703-43fe-bab0-ee25d6261bdc Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2a:a0:78:25:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2a:a0:78:25:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:d6:d9:5b:49:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.795661 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.795766 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.799175 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.799203 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-47.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.799902 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.799911 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.799925 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.801023 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.801720 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804353 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804480 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804509 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804519 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804533 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804542 2576 kubelet.go:398] "Adding apiserver pod source" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804564 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 13:29:19.071728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804937 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.804952 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.807024 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.808334 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.808410 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809006 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809023 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809031 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809039 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809051 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809059 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809067 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809075 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809084 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809091 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809102 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809109 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809856 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.809866 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.813298 2576 server.go:1268] "Started kubelet" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.816364 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.816266 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.816474 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.818088 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.819741 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.822299 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.822324 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.822415 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.822952 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.822954 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.822984 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823065 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823070 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823082 2576 factory.go:55] Registering systemd factory Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823115 2576 factory.go:223] Registration of the systemd container factory successfully Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.823275 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823283 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-47.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823849 2576 factory.go:153] Registering CRI-O factory Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823865 2576 factory.go:223] Registration of the crio container factory successfully Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823929 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823950 2576 factory.go:103] Registering Raw factory Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.823964 2576 manager.go:1196] Started watching for new ooms in manager Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.824002 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 13:29:19.072588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.824372 2576 manager.go:349] Starting recovery of all containers Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.823334 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-47.ec2.internal.18c279a697f4bf39 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-47.ec2.internal,UID:ip-10-0-129-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-47.ec2.internal,},FirstTimestamp:2026-07-15 13:29:18.813069113 +0000 UTC m=+0.330693302,LastTimestamp:2026-07-15 13:29:18.813069113 +0000 UTC m=+0.330693302,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-47.ec2.internal,}" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.830941 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hcdkx" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.835059 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.835065 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.836792 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hcdkx" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.853059 2576 manager.go:354] Recovery completed Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.857198 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.859483 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.859513 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.859523 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.859964 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.859976 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.859984 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.859991 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.860741 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.861015 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.861042 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.861141 2576 policy_none.go:49] "None policy: Start" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.861163 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.861179 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.862205 2576 policy_none.go:47] "Start" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.862310 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-47.ec2.internal.18c279a69ab93582 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-47.ec2.internal,UID:ip-10-0-129-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-47.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-47.ec2.internal,},FirstTimestamp:2026-07-15 13:29:18.859498882 +0000 UTC m=+0.377123074,LastTimestamp:2026-07-15 13:29:18.859498882 +0000 UTC m=+0.377123074,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-47.ec2.internal,}" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.906025 2576 manager.go:337] "Starting Device Plugin manager" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.906211 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.906227 2576 server.go:87] "Starting device plugin registration server" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.906441 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.906452 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.906649 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.906764 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:18.906774 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.907149 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 13:29:19.076614 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:18.907191 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.006853 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.007933 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.007963 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.007978 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.008011 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.016192 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.016213 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-47.ec2.internal\": node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.078121 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.029023 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.088552 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.089622 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.089656 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.089676 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.089717 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.092038 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.129375 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.190552 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"] Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.190622 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.192577 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.192605 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.192614 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.194956 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195123 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195708 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195728 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195730 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195738 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195754 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.195770 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.199513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.199535 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.200253 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.200285 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.200300 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.223180 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-47.ec2.internal\" not found" node="ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.225453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9beb9a962dc749b2a470c1e0a4185fe1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"9beb9a962dc749b2a470c1e0a4185fe1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.225476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9beb9a962dc749b2a470c1e0a4185fe1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"9beb9a962dc749b2a470c1e0a4185fe1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.225500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f903a0bdcc0b6d32ccdde50ff8a5e596-config\") pod \"kube-apiserver-proxy-ip-10-0-129-47.ec2.internal\" (UID: \"f903a0bdcc0b6d32ccdde50ff8a5e596\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.227753 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-47.ec2.internal\" not found" node="ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.232058 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.229813 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.427319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.326651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9beb9a962dc749b2a470c1e0a4185fe1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"9beb9a962dc749b2a470c1e0a4185fe1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.427319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.326688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f903a0bdcc0b6d32ccdde50ff8a5e596-config\") pod \"kube-apiserver-proxy-ip-10-0-129-47.ec2.internal\" (UID: \"f903a0bdcc0b6d32ccdde50ff8a5e596\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.427319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.326713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9beb9a962dc749b2a470c1e0a4185fe1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"9beb9a962dc749b2a470c1e0a4185fe1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.427319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.326756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9beb9a962dc749b2a470c1e0a4185fe1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"9beb9a962dc749b2a470c1e0a4185fe1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.427319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.326766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f903a0bdcc0b6d32ccdde50ff8a5e596-config\") pod \"kube-apiserver-proxy-ip-10-0-129-47.ec2.internal\" (UID: \"f903a0bdcc0b6d32ccdde50ff8a5e596\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.427319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.326766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9beb9a962dc749b2a470c1e0a4185fe1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"9beb9a962dc749b2a470c1e0a4185fe1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.427319 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.329941 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.431218 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.430912 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.586990 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.529676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.586990 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.530855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" Jul 15 13:29:19.586990 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.532058 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.740764 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.633029 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.740764 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.707909 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:29:19.740764 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.733662 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.912997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.763312 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 13:29:19.912997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.763467 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 13:29:19.912997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.763466 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 13:29:19.912997 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.833915 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:19.912997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.845023 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:29:19.912997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.846143 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 13:24:19 +0000 UTC" deadline="2028-01-25 13:46:23.545002664 +0000 UTC" Jul 15 13:29:19.912997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:19.846165 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13416h17m3.698840514s" Jul 15 13:29:20.080599 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:19.934497 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:20.080599 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.035154 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:20.223289 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.135835 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:20.389708 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.236916 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:20.389708 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:20.301657 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf903a0bdcc0b6d32ccdde50ff8a5e596.slice/crio-34ad3f96e8559b92ebb6bf2f3c84eea81bd29b5b2f8d4feff90032b884d48845 WatchSource:0}: Error finding container 34ad3f96e8559b92ebb6bf2f3c84eea81bd29b5b2f8d4feff90032b884d48845: Status 404 returned error can't find the container with id 34ad3f96e8559b92ebb6bf2f3c84eea81bd29b5b2f8d4feff90032b884d48845 Jul 15 13:29:20.389708 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:20.301993 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9beb9a962dc749b2a470c1e0a4185fe1.slice/crio-6979e612bdb11e38d6393a7f86e5a47c488a8490b555581d8cef9f652810933e WatchSource:0}: Error finding container 6979e612bdb11e38d6393a7f86e5a47c488a8490b555581d8cef9f652810933e: Status 404 returned error can't find the container with id 6979e612bdb11e38d6393a7f86e5a47c488a8490b555581d8cef9f652810933e Jul 15 13:29:20.389708 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.307899 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:29:20.389708 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.337658 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found" Jul 15 13:29:20.554373 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.418562 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:29:20.554373 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.423434 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" Jul 15 13:29:20.554373 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.435156 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 13:29:20.554373 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.436170 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" Jul 15 13:29:20.554373 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.445130 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 13:29:20.671358 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.660056 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.805747 2576 apiserver.go:52] "Watching apiserver" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.812790 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.813171 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-6mprk","openshift-multus/network-metrics-daemon-b62v4","openshift-ovn-kubernetes/ovnkube-node-rps9b","kube-system/global-pull-secret-syncer-tkwn8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7","openshift-cluster-node-tuning-operator/tuned-48d6s","openshift-image-registry/node-ca-hkm5q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal","openshift-multus/multus-bw8kv","openshift-network-diagnostics/network-check-target-rbsfv","openshift-network-operator/iptables-alerter-7r9tz","kube-system/konnectivity-agent-q657c","kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal","openshift-dns/node-resolver-spfvj"] Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.815808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.818014 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.818026 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z82q7\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.818127 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.818279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.818731 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.820650 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.820779 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.820866 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.820914 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-blkbb\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.820784 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.822980 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823202 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823310 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nchvk\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823514 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823578 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823673 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.823662 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.823862 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.825731 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.827840 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.833464 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qwzhg\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.834118 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.834253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.835628 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.836284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-htw5m\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.836506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:29:20.854452 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.836845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.837143 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.838858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lczg2\"" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.838994 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.839039 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.839092 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.839113 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.839379 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.839713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b16723e3-98fe-44ca-b9c4-c33598b42571-host-slash\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-slash\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-ovn\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-device-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-sys-fs\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-var-lib-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-env-overrides\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-socket-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5qf\" (UniqueName: \"kubernetes.io/projected/d29ac1b1-707e-4791-97f3-62acd643c83d-kube-api-access-wf5qf\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-kubelet\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrz2s\" (UniqueName: \"kubernetes.io/projected/944ce9ec-3ae4-431a-8f68-c6c720fc7613-kube-api-access-lrz2s\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.856162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eafabcae-943c-4420-b244-a96aa27bedb9-kubelet-config\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-registration-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d29ac1b1-707e-4791-97f3-62acd643c83d-host\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpzz\" (UniqueName: \"kubernetes.io/projected/b16723e3-98fe-44ca-b9c4-c33598b42571-kube-api-access-mdpzz\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovnkube-script-lib\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-log-socket\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-systemd-units\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-etc-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-node-log\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-cni-bin\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.840986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovnkube-config\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d29ac1b1-707e-4791-97f3-62acd643c83d-serviceca\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-run-ovn-kubernetes\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-run-netns\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-systemd\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-cni-netd\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovn-node-metrics-cert\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eafabcae-943c-4420-b244-a96aa27bedb9-dbus\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv49v\" (UniqueName: \"kubernetes.io/projected/7bf07266-916e-4589-a369-a353d25ec86b-kube-api-access-tv49v\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.841223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b16723e3-98fe-44ca-b9c4-c33598b42571-iptables-alerter-script\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.842083 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.842414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.842476 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.842415 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.843046 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qk7h9\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.844637 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.844697 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.847253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.849297 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.849349 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8lnwl\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.849516 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.849752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.853841 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.854084 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7pngg\"" Jul 15 13:29:20.858847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.854271 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 13:29:20.861846 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.860247 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9vkq" Jul 15 13:29:20.868811 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.868788 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9vkq" Jul 15 13:29:20.924520 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.924437 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 13:29:20.941721 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovnkube-script-lib\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.941841 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.941841 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-log-socket\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.941841 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:20.941841 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-cni-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.941978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-k8s-cni-cncf-io\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.941978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.941978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6lj\" (UniqueName: \"kubernetes.io/projected/03c7b4c8-4578-4224-b998-2c729967387f-kube-api-access-wj6lj\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.941978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-systemd-units\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.941978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-etc-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.941978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-node-log\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.941996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-cni-bin\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46tn\" (UniqueName: \"kubernetes.io/projected/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-kube-api-access-z46tn\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-cni-bin\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942075 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-multus-certs\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-systemd-units\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-system-cni-dir\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d29ac1b1-707e-4791-97f3-62acd643c83d-serviceca\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-run-ovn-kubernetes\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942188 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-etc-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53adf7c9-4b8b-4f34-aba1-fffe96e1b434-konnectivity-ca\") pod \"konnectivity-agent-q657c\" (UID: \"53adf7c9-4b8b-4f34-aba1-fffe96e1b434\") " pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-node-log\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-modprobe-d\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-run-netns\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942280 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-cni-bin\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovn-node-metrics-cert\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.943142 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-log-socket\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-cnibin\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-os-release\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70669082-cd5b-4f86-9466-9a95cc20a578-cni-binary-copy\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysctl-d\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-host\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.942498 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:20.942583 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret podName:eafabcae-943c-4420-b244-a96aa27bedb9 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:21.442558942 +0000 UTC m=+2.960183138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret") pod "global-pull-secret-syncer-tkwn8" (UID: "eafabcae-943c-4420-b244-a96aa27bedb9") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-run-ovn-kubernetes\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-run-netns\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhblk\" (UniqueName: \"kubernetes.io/projected/45cc404e-25d4-4535-8159-5621f2aca1f1-kube-api-access-rhblk\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b16723e3-98fe-44ca-b9c4-c33598b42571-iptables-alerter-script\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b16723e3-98fe-44ca-b9c4-c33598b42571-host-slash\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-ovn\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-device-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942824 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 13:29:20.944069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-cnibin\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-kubernetes\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-var-lib-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-env-overrides\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-ovn\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.942996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b16723e3-98fe-44ca-b9c4-c33598b42571-host-slash\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d29ac1b1-707e-4791-97f3-62acd643c83d-serviceca\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-device-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-var-lib-openvswitch\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b16723e3-98fe-44ca-b9c4-c33598b42571-iptables-alerter-script\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-socket-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-socket-dir-parent\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943285 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-socket-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysconfig\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-systemd\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.944968 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5qf\" (UniqueName: \"kubernetes.io/projected/d29ac1b1-707e-4791-97f3-62acd643c83d-kube-api-access-wf5qf\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovnkube-script-lib\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-env-overrides\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-netns\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943448 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-cni-multus\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysctl-conf\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-tuned\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d29ac1b1-707e-4791-97f3-62acd643c83d-host\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-tmp-dir\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-etc-kubernetes\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d29ac1b1-707e-4791-97f3-62acd643c83d-host\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-sys\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-system-cni-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70669082-cd5b-4f86-9466-9a95cc20a578-multus-daemon-config\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovnkube-config\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.945535 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-hosts-file\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-run\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943936 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-systemd\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.943979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-cni-netd\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eafabcae-943c-4420-b244-a96aa27bedb9-dbus\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-run-systemd\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv49v\" (UniqueName: \"kubernetes.io/projected/7bf07266-916e-4589-a369-a353d25ec86b-kube-api-access-tv49v\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-conf-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-slash\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-sys-fs\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r677z\" (UniqueName: \"kubernetes.io/projected/62441196-0328-49dc-aeb8-714c0357b917-kube-api-access-r677z\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-os-release\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53adf7c9-4b8b-4f34-aba1-fffe96e1b434-agent-certs\") pod \"konnectivity-agent-q657c\" (UID: \"53adf7c9-4b8b-4f34-aba1-fffe96e1b434\") " pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:20.946080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-lib-modules\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-var-lib-kubelet\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eafabcae-943c-4420-b244-a96aa27bedb9-dbus\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-kubelet\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovnkube-config\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-cni-netd\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-sys-fs\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrz2s\" (UniqueName: \"kubernetes.io/projected/944ce9ec-3ae4-431a-8f68-c6c720fc7613-kube-api-access-lrz2s\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eafabcae-943c-4420-b244-a96aa27bedb9-kubelet-config\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944391 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-slash\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-registration-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-hostroot\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpzz\" (UniqueName: \"kubernetes.io/projected/b16723e3-98fe-44ca-b9c4-c33598b42571-kube-api-access-mdpzz\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-kubelet\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvt2z\" (UniqueName: \"kubernetes.io/projected/70669082-cd5b-4f86-9466-9a95cc20a578-kube-api-access-vvt2z\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45cc404e-25d4-4535-8159-5621f2aca1f1-tmp\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:20.946815 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-registration-dir\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.947631 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/944ce9ec-3ae4-431a-8f68-c6c720fc7613-host-kubelet\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.947631 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bf07266-916e-4589-a369-a353d25ec86b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:20.947631 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.944801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eafabcae-943c-4420-b244-a96aa27bedb9-kubelet-config\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:20.947817 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.947802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/944ce9ec-3ae4-431a-8f68-c6c720fc7613-ovn-node-metrics-cert\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.953989 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.953966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5qf\" (UniqueName: \"kubernetes.io/projected/d29ac1b1-707e-4791-97f3-62acd643c83d-kube-api-access-wf5qf\") pod \"node-ca-hkm5q\" (UID: \"d29ac1b1-707e-4791-97f3-62acd643c83d\") " pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:20.954363 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.954341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrz2s\" (UniqueName: \"kubernetes.io/projected/944ce9ec-3ae4-431a-8f68-c6c720fc7613-kube-api-access-lrz2s\") pod \"ovnkube-node-rps9b\" (UID: \"944ce9ec-3ae4-431a-8f68-c6c720fc7613\") " pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:20.956274 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.956231 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpzz\" (UniqueName: \"kubernetes.io/projected/b16723e3-98fe-44ca-b9c4-c33598b42571-kube-api-access-mdpzz\") pod \"iptables-alerter-7r9tz\" (UID: \"b16723e3-98fe-44ca-b9c4-c33598b42571\") " pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:20.957537 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:20.957515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv49v\" (UniqueName: \"kubernetes.io/projected/7bf07266-916e-4589-a369-a353d25ec86b-kube-api-access-tv49v\") pod \"aws-ebs-csi-driver-node-pnzs7\" (UID: \"7bf07266-916e-4589-a369-a353d25ec86b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:21.045344 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-hosts-file\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:21.045531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-run\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.045531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-conf-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.045531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:21.045708 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r677z\" (UniqueName: \"kubernetes.io/projected/62441196-0328-49dc-aeb8-714c0357b917-kube-api-access-r677z\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:21.045708 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-os-release\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.045708 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53adf7c9-4b8b-4f34-aba1-fffe96e1b434-agent-certs\") pod \"konnectivity-agent-q657c\" (UID: \"53adf7c9-4b8b-4f34-aba1-fffe96e1b434\") " pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:21.045708 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-lib-modules\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.045903 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-lib-modules\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.045903 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-conf-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.045903 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-run\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.046051 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-var-lib-kubelet\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.046051 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-hosts-file\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:21.046051 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-os-release\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.046051 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-hostroot\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046051 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-var-lib-kubelet\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.046051 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.045957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-hostroot\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-kubelet\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvt2z\" (UniqueName: \"kubernetes.io/projected/70669082-cd5b-4f86-9466-9a95cc20a578-kube-api-access-vvt2z\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45cc404e-25d4-4535-8159-5621f2aca1f1-tmp\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-cni-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-k8s-cni-cncf-io\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6lj\" (UniqueName: \"kubernetes.io/projected/03c7b4c8-4578-4224-b998-2c729967387f-kube-api-access-wj6lj\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.046337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46tn\" (UniqueName: \"kubernetes.io/projected/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-kube-api-access-z46tn\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-cni-bin\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.046390 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-multus-certs\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.046473 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs podName:62441196-0328-49dc-aeb8-714c0357b917 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:21.546431464 +0000 UTC m=+3.064055643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs") pod "network-metrics-daemon-b62v4" (UID: "62441196-0328-49dc-aeb8-714c0357b917") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-multus-certs\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-system-cni-dir\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-cni-bin\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53adf7c9-4b8b-4f34-aba1-fffe96e1b434-konnectivity-ca\") pod \"konnectivity-agent-q657c\" (UID: \"53adf7c9-4b8b-4f34-aba1-fffe96e1b434\") " pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-modprobe-d\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-cnibin\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-os-release\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046695 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70669082-cd5b-4f86-9466-9a95cc20a578-cni-binary-copy\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysctl-d\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.046773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-host\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.049924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70669082-cd5b-4f86-9466-9a95cc20a578-cni-binary-copy\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-host\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysctl-d\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050102 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-kubelet\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-system-cni-dir\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.046798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhblk\" (UniqueName: \"kubernetes.io/projected/45cc404e-25d4-4535-8159-5621f2aca1f1-kube-api-access-rhblk\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-cnibin\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-kubernetes\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-socket-dir-parent\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysconfig\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050596 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/53adf7c9-4b8b-4f34-aba1-fffe96e1b434-agent-certs\") pod \"konnectivity-agent-q657c\" (UID: \"53adf7c9-4b8b-4f34-aba1-fffe96e1b434\") " pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-systemd\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-netns\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-systemd\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-cni-multus\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050722 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-netns\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050732 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/53adf7c9-4b8b-4f34-aba1-fffe96e1b434-konnectivity-ca\") pod \"konnectivity-agent-q657c\" (UID: \"53adf7c9-4b8b-4f34-aba1-fffe96e1b434\") " pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:21.054436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-cnibin\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050856 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysconfig\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysctl-conf\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-tuned\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-kubernetes\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.050947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-tmp-dir\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-etc-kubernetes\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-sysctl-conf\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-sys\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051122 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-var-lib-cni-multus\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-system-cni-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70669082-cd5b-4f86-9466-9a95cc20a578-multus-daemon-config\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-system-cni-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-sys\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-socket-dir-parent\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.055222 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-tmp-dir\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-modprobe-d\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-os-release\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-multus-cni-dir\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.051892 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03c7b4c8-4578-4224-b998-2c729967387f-cnibin\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.052170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.052243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-etc-kubernetes\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.052405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.052457 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03c7b4c8-4578-4224-b998-2c729967387f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.052613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70669082-cd5b-4f86-9466-9a95cc20a578-host-run-k8s-cni-cncf-io\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.056036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.052764 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70669082-cd5b-4f86-9466-9a95cc20a578-multus-daemon-config\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.058027 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.056706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/45cc404e-25d4-4535-8159-5621f2aca1f1-etc-tuned\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.058027 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.057274 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:29:21.058027 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.057295 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:29:21.058027 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.057310 2576 projected.go:196] Error preparing data for projected volume kube-api-access-hglq2 for pod openshift-network-diagnostics/network-check-target-rbsfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:21.058027 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.057382 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2 podName:0609e1f4-a178-4ed3-aa1c-22c27d375e87 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:21.557365347 +0000 UTC m=+3.074989550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hglq2" (UniqueName: "kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2") pod "network-check-target-rbsfv" (UID: "0609e1f4-a178-4ed3-aa1c-22c27d375e87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:21.058357 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.058199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45cc404e-25d4-4535-8159-5621f2aca1f1-tmp\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.062374 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.062351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r677z\" (UniqueName: \"kubernetes.io/projected/62441196-0328-49dc-aeb8-714c0357b917-kube-api-access-r677z\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:21.064315 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.064246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvt2z\" (UniqueName: \"kubernetes.io/projected/70669082-cd5b-4f86-9466-9a95cc20a578-kube-api-access-vvt2z\") pod \"multus-bw8kv\" (UID: \"70669082-cd5b-4f86-9466-9a95cc20a578\") " pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.064315 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.064272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhblk\" (UniqueName: \"kubernetes.io/projected/45cc404e-25d4-4535-8159-5621f2aca1f1-kube-api-access-rhblk\") pod \"tuned-48d6s\" (UID: \"45cc404e-25d4-4535-8159-5621f2aca1f1\") " pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.064741 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.064720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6lj\" (UniqueName: \"kubernetes.io/projected/03c7b4c8-4578-4224-b998-2c729967387f-kube-api-access-wj6lj\") pod \"multus-additional-cni-plugins-6mprk\" (UID: \"03c7b4c8-4578-4224-b998-2c729967387f\") " pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.066768 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.066743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46tn\" (UniqueName: \"kubernetes.io/projected/77907d2e-8cca-44e7-a03b-8b34bf3bd2bf-kube-api-access-z46tn\") pod \"node-resolver-spfvj\" (UID: \"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf\") " pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:21.096912 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.096857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" event={"ID":"9beb9a962dc749b2a470c1e0a4185fe1","Type":"ContainerStarted","Data":"6979e612bdb11e38d6393a7f86e5a47c488a8490b555581d8cef9f652810933e"} Jul 15 13:29:21.099110 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.098732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" event={"ID":"f903a0bdcc0b6d32ccdde50ff8a5e596","Type":"ContainerStarted","Data":"34ad3f96e8559b92ebb6bf2f3c84eea81bd29b5b2f8d4feff90032b884d48845"} Jul 15 13:29:21.139615 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.139382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkm5q" Jul 15 13:29:21.152239 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.152208 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29ac1b1_707e_4791_97f3_62acd643c83d.slice/crio-18b4d0d0e003a105e2fe32ac64e6a645e8a50ccf587cba5e77146a9c7c72ff9e WatchSource:0}: Error finding container 18b4d0d0e003a105e2fe32ac64e6a645e8a50ccf587cba5e77146a9c7c72ff9e: Status 404 returned error can't find the container with id 18b4d0d0e003a105e2fe32ac64e6a645e8a50ccf587cba5e77146a9c7c72ff9e Jul 15 13:29:21.163130 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.162928 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7r9tz" Jul 15 13:29:21.169351 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.169186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:21.174029 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.174007 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" Jul 15 13:29:21.174675 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.174584 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16723e3_98fe_44ca_b9c4_c33598b42571.slice/crio-5c6c933a0db1e9f03a8ab89e93879cc4f5ded3d7b37d1ac22bc261e47053a283 WatchSource:0}: Error finding container 5c6c933a0db1e9f03a8ab89e93879cc4f5ded3d7b37d1ac22bc261e47053a283: Status 404 returned error can't find the container with id 5c6c933a0db1e9f03a8ab89e93879cc4f5ded3d7b37d1ac22bc261e47053a283 Jul 15 13:29:21.182832 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.182669 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-48d6s" Jul 15 13:29:21.183026 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.183003 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944ce9ec_3ae4_431a_8f68_c6c720fc7613.slice/crio-175386ba871773381a754fa82fda936e6580eee3fe93422104ae482ec5a19cf6 WatchSource:0}: Error finding container 175386ba871773381a754fa82fda936e6580eee3fe93422104ae482ec5a19cf6: Status 404 returned error can't find the container with id 175386ba871773381a754fa82fda936e6580eee3fe93422104ae482ec5a19cf6 Jul 15 13:29:21.190208 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.187419 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf07266_916e_4589_a369_a353d25ec86b.slice/crio-8b79a63a74806c13d9133c1cca3a2d5e1f750fb258f22efd408018b27c65f0d2 WatchSource:0}: Error finding container 8b79a63a74806c13d9133c1cca3a2d5e1f750fb258f22efd408018b27c65f0d2: Status 404 returned error can't find the container with id 8b79a63a74806c13d9133c1cca3a2d5e1f750fb258f22efd408018b27c65f0d2 Jul 15 13:29:21.190688 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.190629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw8kv" Jul 15 13:29:21.193252 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.193225 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cc404e_25d4_4535_8159_5621f2aca1f1.slice/crio-deaa889200b7bff544787bbd2befc5b79817e6367327ceb50d749b967221ee3e WatchSource:0}: Error finding container deaa889200b7bff544787bbd2befc5b79817e6367327ceb50d749b967221ee3e: Status 404 returned error can't find the container with id deaa889200b7bff544787bbd2befc5b79817e6367327ceb50d749b967221ee3e Jul 15 13:29:21.199591 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.199566 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spfvj" Jul 15 13:29:21.204994 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.204963 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70669082_cd5b_4f86_9466_9a95cc20a578.slice/crio-5783bf8462080053b035ec5bafc20121bf8bf1d11485b2ade8a73a6052daa909 WatchSource:0}: Error finding container 5783bf8462080053b035ec5bafc20121bf8bf1d11485b2ade8a73a6052daa909: Status 404 returned error can't find the container with id 5783bf8462080053b035ec5bafc20121bf8bf1d11485b2ade8a73a6052daa909 Jul 15 13:29:21.205752 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.205734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6mprk" Jul 15 13:29:21.212409 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.212391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:21.213048 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.213019 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77907d2e_8cca_44e7_a03b_8b34bf3bd2bf.slice/crio-c2d1e7247abc6e1b2f6e926436ee83ae2de89f68d9fc8829efd62b866085acf3 WatchSource:0}: Error finding container c2d1e7247abc6e1b2f6e926436ee83ae2de89f68d9fc8829efd62b866085acf3: Status 404 returned error can't find the container with id c2d1e7247abc6e1b2f6e926436ee83ae2de89f68d9fc8829efd62b866085acf3 Jul 15 13:29:21.219379 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.219353 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c7b4c8_4578_4224_b998_2c729967387f.slice/crio-f4ba2f684f76a94b4fc3c4e31da0fe02f88f8feb07c67fe7dc37eefebe03380e WatchSource:0}: Error finding container f4ba2f684f76a94b4fc3c4e31da0fe02f88f8feb07c67fe7dc37eefebe03380e: Status 404 returned error can't find the container with id f4ba2f684f76a94b4fc3c4e31da0fe02f88f8feb07c67fe7dc37eefebe03380e Jul 15 13:29:21.225994 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:29:21.225962 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53adf7c9_4b8b_4f34_aba1_fffe96e1b434.slice/crio-42f204eea35c83f574826e193a13409b1f3f90055ff3c4b8161afb1b02686d23 WatchSource:0}: Error finding container 42f204eea35c83f574826e193a13409b1f3f90055ff3c4b8161afb1b02686d23: Status 404 returned error can't find the container with id 42f204eea35c83f574826e193a13409b1f3f90055ff3c4b8161afb1b02686d23 Jul 15 13:29:21.454471 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.454364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:21.454645 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.454538 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:21.454645 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.454610 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret podName:eafabcae-943c-4420-b244-a96aa27bedb9 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:22.454592893 +0000 UTC m=+3.972217076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret") pod "global-pull-secret-syncer-tkwn8" (UID: "eafabcae-943c-4420-b244-a96aa27bedb9") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:21.555036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.554994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:21.555225 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.555142 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:21.555287 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.555226 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs podName:62441196-0328-49dc-aeb8-714c0357b917 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:22.555203416 +0000 UTC m=+4.072827605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs") pod "network-metrics-daemon-b62v4" (UID: "62441196-0328-49dc-aeb8-714c0357b917") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:21.656192 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.656120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:21.656389 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.656270 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:29:21.656389 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.656297 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:29:21.656389 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.656311 2576 projected.go:196] Error preparing data for projected volume kube-api-access-hglq2 for pod openshift-network-diagnostics/network-check-target-rbsfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:21.656389 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:21.656385 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2 podName:0609e1f4-a178-4ed3-aa1c-22c27d375e87 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:22.656365521 +0000 UTC m=+4.173989717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hglq2" (UniqueName: "kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2") pod "network-check-target-rbsfv" (UID: "0609e1f4-a178-4ed3-aa1c-22c27d375e87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:21.872547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.872499 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 13:24:21 +0000 UTC" deadline="2028-01-02 02:51:20.22928341 +0000 UTC" Jul 15 13:29:21.872734 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:21.872545 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12853h21m58.356743636s" Jul 15 13:29:22.090932 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.090900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:22.091659 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.091039 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:22.104105 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.104029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q657c" event={"ID":"53adf7c9-4b8b-4f34-aba1-fffe96e1b434","Type":"ContainerStarted","Data":"42f204eea35c83f574826e193a13409b1f3f90055ff3c4b8161afb1b02686d23"} Jul 15 13:29:22.106103 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.106054 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spfvj" event={"ID":"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf","Type":"ContainerStarted","Data":"c2d1e7247abc6e1b2f6e926436ee83ae2de89f68d9fc8829efd62b866085acf3"} Jul 15 13:29:22.108793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.108722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-48d6s" event={"ID":"45cc404e-25d4-4535-8159-5621f2aca1f1","Type":"ContainerStarted","Data":"deaa889200b7bff544787bbd2befc5b79817e6367327ceb50d749b967221ee3e"} Jul 15 13:29:22.113085 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.113060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkm5q" event={"ID":"d29ac1b1-707e-4791-97f3-62acd643c83d","Type":"ContainerStarted","Data":"18b4d0d0e003a105e2fe32ac64e6a645e8a50ccf587cba5e77146a9c7c72ff9e"} Jul 15 13:29:22.115884 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.115837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerStarted","Data":"f4ba2f684f76a94b4fc3c4e31da0fe02f88f8feb07c67fe7dc37eefebe03380e"} Jul 15 13:29:22.117833 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.117811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw8kv" event={"ID":"70669082-cd5b-4f86-9466-9a95cc20a578","Type":"ContainerStarted","Data":"5783bf8462080053b035ec5bafc20121bf8bf1d11485b2ade8a73a6052daa909"} Jul 15 13:29:22.119190 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.119096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" event={"ID":"7bf07266-916e-4589-a369-a353d25ec86b","Type":"ContainerStarted","Data":"8b79a63a74806c13d9133c1cca3a2d5e1f750fb258f22efd408018b27c65f0d2"} Jul 15 13:29:22.121017 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.120981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"175386ba871773381a754fa82fda936e6580eee3fe93422104ae482ec5a19cf6"} Jul 15 13:29:22.122644 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.122585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7r9tz" event={"ID":"b16723e3-98fe-44ca-b9c4-c33598b42571","Type":"ContainerStarted","Data":"5c6c933a0db1e9f03a8ab89e93879cc4f5ded3d7b37d1ac22bc261e47053a283"} Jul 15 13:29:22.462868 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.462777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:22.463032 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.462966 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:22.463100 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.463040 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret podName:eafabcae-943c-4420-b244-a96aa27bedb9 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:24.463022521 +0000 UTC m=+5.980646725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret") pod "global-pull-secret-syncer-tkwn8" (UID: "eafabcae-943c-4420-b244-a96aa27bedb9") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:22.564120 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.564079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:22.564316 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.564299 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:22.564391 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.564375 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs podName:62441196-0328-49dc-aeb8-714c0357b917 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:24.564351998 +0000 UTC m=+6.081976191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs") pod "network-metrics-daemon-b62v4" (UID: "62441196-0328-49dc-aeb8-714c0357b917") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:22.664876 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.664835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:22.665083 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.665064 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:29:22.665153 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.665092 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:29:22.665153 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.665106 2576 projected.go:196] Error preparing data for projected volume kube-api-access-hglq2 for pod openshift-network-diagnostics/network-check-target-rbsfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:22.665255 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:22.665171 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2 podName:0609e1f4-a178-4ed3-aa1c-22c27d375e87 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:24.665150561 +0000 UTC m=+6.182774741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hglq2" (UniqueName: "kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2") pod "network-check-target-rbsfv" (UID: "0609e1f4-a178-4ed3-aa1c-22c27d375e87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:22.671802 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.671771 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:29:22.873797 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.873756 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 13:24:21 +0000 UTC" deadline="2028-03-19 17:51:18.119560554 +0000 UTC" Jul 15 13:29:22.873797 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:22.873795 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14716h21m55.245770015s" Jul 15 13:29:23.091538 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:23.091501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:23.091927 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:23.091656 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:23.092520 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:23.092495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:23.092641 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:23.092618 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:24.090371 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:24.090333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:24.090547 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.090503 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:24.479947 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:24.479905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:24.480440 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.480136 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:24.480440 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.480230 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret podName:eafabcae-943c-4420-b244-a96aa27bedb9 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:28.480210144 +0000 UTC m=+9.997834338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret") pod "global-pull-secret-syncer-tkwn8" (UID: "eafabcae-943c-4420-b244-a96aa27bedb9") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:24.581266 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:24.581224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:24.581458 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.581422 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:24.581586 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.581508 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs podName:62441196-0328-49dc-aeb8-714c0357b917 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:28.581488432 +0000 UTC m=+10.099112613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs") pod "network-metrics-daemon-b62v4" (UID: "62441196-0328-49dc-aeb8-714c0357b917") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:24.682336 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:24.682294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:24.682571 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.682551 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:29:24.682640 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.682584 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:29:24.682640 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.682600 2576 projected.go:196] Error preparing data for projected volume kube-api-access-hglq2 for pod openshift-network-diagnostics/network-check-target-rbsfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:24.682762 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:24.682667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2 podName:0609e1f4-a178-4ed3-aa1c-22c27d375e87 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:28.682646263 +0000 UTC m=+10.200270446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hglq2" (UniqueName: "kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2") pod "network-check-target-rbsfv" (UID: "0609e1f4-a178-4ed3-aa1c-22c27d375e87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:25.093835 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:25.093491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:25.093995 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:25.093906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:25.094061 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:25.093903 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:25.094061 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:25.094033 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:26.090504 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:26.090469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:26.090977 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:26.090594 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:26.137557 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:26.137511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" event={"ID":"9beb9a962dc749b2a470c1e0a4185fe1","Type":"ContainerStarted","Data":"a524778d43ad42fd928ab8b16b93deee8f3a9b1cfbcd622e93bf8b07665ddf56"} Jul 15 13:29:27.090800 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:27.090721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:27.091219 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:27.090879 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:27.091219 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:27.090926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:27.091219 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:27.091038 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:27.141517 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:27.141477 2576 generic.go:356] "Generic (PLEG): container finished" podID="9beb9a962dc749b2a470c1e0a4185fe1" containerID="a524778d43ad42fd928ab8b16b93deee8f3a9b1cfbcd622e93bf8b07665ddf56" exitCode=0 Jul 15 13:29:27.141654 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:27.141535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" event={"ID":"9beb9a962dc749b2a470c1e0a4185fe1","Type":"ContainerDied","Data":"a524778d43ad42fd928ab8b16b93deee8f3a9b1cfbcd622e93bf8b07665ddf56"} Jul 15 13:29:28.090038 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:28.089950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:28.090197 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.090082 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:28.516035 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:28.515993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:28.516461 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.516180 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:28.516461 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.516274 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret podName:eafabcae-943c-4420-b244-a96aa27bedb9 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:36.516250561 +0000 UTC m=+18.033874787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret") pod "global-pull-secret-syncer-tkwn8" (UID: "eafabcae-943c-4420-b244-a96aa27bedb9") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:28.617020 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:28.616986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:28.617203 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.617155 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:28.617255 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.617237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs podName:62441196-0328-49dc-aeb8-714c0357b917 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:36.617217359 +0000 UTC m=+18.134841549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs") pod "network-metrics-daemon-b62v4" (UID: "62441196-0328-49dc-aeb8-714c0357b917") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:28.718004 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:28.717965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:28.718197 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.718170 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:29:28.718197 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.718193 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:29:28.718318 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.718206 2576 projected.go:196] Error preparing data for projected volume kube-api-access-hglq2 for pod openshift-network-diagnostics/network-check-target-rbsfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:28.718318 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:28.718266 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2 podName:0609e1f4-a178-4ed3-aa1c-22c27d375e87 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:36.71824923 +0000 UTC m=+18.235873410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hglq2" (UniqueName: "kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2") pod "network-check-target-rbsfv" (UID: "0609e1f4-a178-4ed3-aa1c-22c27d375e87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:29.091503 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:29.091468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:29.092477 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:29.092414 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:29.092610 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:29.092489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:29.092610 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:29.092583 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:30.090091 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:30.090064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:30.090542 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:30.090169 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:31.090614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:31.090570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:31.091098 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:31.090705 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:31.091098 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:31.090755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:31.091098 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:31.090874 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:32.090108 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:32.090027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:32.090250 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:32.090144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:33.090492 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:33.090459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:33.090936 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:33.090500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:33.090936 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:33.090596 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:33.090936 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:33.090724 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:34.090014 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:34.089975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:34.090165 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:34.090103 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:35.090471 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:35.090418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:35.090891 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:35.090421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:35.090891 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:35.090555 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:35.090891 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:35.090619 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:36.090119 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:36.090087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:36.090342 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.090207 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:36.578717 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:36.578680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:36.579158 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.578839 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:36.579158 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.578913 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret podName:eafabcae-943c-4420-b244-a96aa27bedb9 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:52.578894916 +0000 UTC m=+34.096519098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret") pod "global-pull-secret-syncer-tkwn8" (UID: "eafabcae-943c-4420-b244-a96aa27bedb9") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:36.679409 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:36.679362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:36.679570 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.679520 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:36.679622 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.679597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs podName:62441196-0328-49dc-aeb8-714c0357b917 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:52.67957508 +0000 UTC m=+34.197199259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs") pod "network-metrics-daemon-b62v4" (UID: "62441196-0328-49dc-aeb8-714c0357b917") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:36.779991 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:36.779957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:36.780169 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.780092 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:29:36.780169 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.780110 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:29:36.780169 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.780120 2576 projected.go:196] Error preparing data for projected volume kube-api-access-hglq2 for pod openshift-network-diagnostics/network-check-target-rbsfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:36.780330 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:36.780176 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2 podName:0609e1f4-a178-4ed3-aa1c-22c27d375e87 nodeName:}" failed. No retries permitted until 2026-07-15 13:29:52.780161808 +0000 UTC m=+34.297785989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hglq2" (UniqueName: "kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2") pod "network-check-target-rbsfv" (UID: "0609e1f4-a178-4ed3-aa1c-22c27d375e87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:37.091170 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:37.090746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:37.091170 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:37.090777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:37.091170 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:37.090865 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:37.091170 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:37.090980 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:38.090127 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:38.090094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:38.090552 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:38.090216 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:39.090893 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:39.090862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:39.091340 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:39.090990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:39.091340 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:39.091103 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:39.091340 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:39.090972 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:40.090419 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.090248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:40.090541 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:40.090522 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:40.166344 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.165948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" event={"ID":"9beb9a962dc749b2a470c1e0a4185fe1","Type":"ContainerStarted","Data":"6f009f96a9904b25cbe917be7768178d491eb127c85ed0342009dbdff3b47418"} Jul 15 13:29:40.167835 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.167490 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerStarted","Data":"084d6e8f84870c3ab05ba306ae322f1f4c08c7a51f2b82b55b05782a02de3c53"} Jul 15 13:29:40.169245 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.169217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" event={"ID":"7bf07266-916e-4589-a369-a353d25ec86b","Type":"ContainerStarted","Data":"c5006ccd46627924f8483de2a4367c001a803eda046322c35c7807f786acc0e4"} Jul 15 13:29:40.171483 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.171461 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:29:40.171905 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.171880 2576 generic.go:356] "Generic (PLEG): container finished" podID="944ce9ec-3ae4-431a-8f68-c6c720fc7613" containerID="30b5ad8c30a246868c66a0273b6ef16b86024a99abecd105644f2425e782a556" exitCode=1 Jul 15 13:29:40.172081 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.171950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerDied","Data":"30b5ad8c30a246868c66a0273b6ef16b86024a99abecd105644f2425e782a556"} Jul 15 13:29:40.172159 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.172149 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"98c2a892ef3636e16e45bd7bf8494d0b7563faf6f0dc5d986cca186434936a17"} Jul 15 13:29:40.173999 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.173971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" event={"ID":"f903a0bdcc0b6d32ccdde50ff8a5e596","Type":"ContainerStarted","Data":"701960ab75a01a61c24f0e9894898d344a0581f1d790a86be3aeff9a3c5a8d25"} Jul 15 13:29:40.176647 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.176621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q657c" event={"ID":"53adf7c9-4b8b-4f34-aba1-fffe96e1b434","Type":"ContainerStarted","Data":"cb95ced0e93351149511327abd47e29ce06f91dc0367841edfbac7d6dadc7223"} Jul 15 13:29:40.178175 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.178156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spfvj" event={"ID":"77907d2e-8cca-44e7-a03b-8b34bf3bd2bf","Type":"ContainerStarted","Data":"34d36293914adeca37b77fc72f1a640a6299e12951c800cd50959b304105cef6"} Jul 15 13:29:40.179821 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.179797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-48d6s" event={"ID":"45cc404e-25d4-4535-8159-5621f2aca1f1","Type":"ContainerStarted","Data":"a6ee05d0bc2973be19586d93e1db7a6adf6e1801995eddd01968a8699117fa4e"} Jul 15 13:29:40.181551 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.181530 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkm5q" event={"ID":"d29ac1b1-707e-4791-97f3-62acd643c83d","Type":"ContainerStarted","Data":"549d66ef2a75df22b8dd560a27f32eec4ea9326f034cca36e4031d1bca73fd0a"} Jul 15 13:29:40.182727 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.182676 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" podStartSLOduration=20.182660228 podStartE2EDuration="20.182660228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:40.181884177 +0000 UTC m=+21.699508397" watchObservedRunningTime="2026-07-15 13:29:40.182660228 +0000 UTC m=+21.700284431" Jul 15 13:29:40.219653 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.219614 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q657c" podStartSLOduration=2.7671940790000003 podStartE2EDuration="21.219596439s" totalImagesPullingTime="18.45240236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:40.219389706 +0000 UTC m=+21.737013904" watchObservedRunningTime="2026-07-15 13:29:40.219596439 +0000 UTC m=+21.737220638" Jul 15 13:29:40.251176 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.251052 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-48d6s" podStartSLOduration=2.668069714 podStartE2EDuration="21.251037849s" totalImagesPullingTime="18.582968135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:40.25070614 +0000 UTC m=+21.768330338" watchObservedRunningTime="2026-07-15 13:29:40.251037849 +0000 UTC m=+21.768662051" Jul 15 13:29:40.251496 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.251471 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" podStartSLOduration=20.251463206 podStartE2EDuration="20.251463206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:40.233588012 +0000 UTC m=+21.751212212" watchObservedRunningTime="2026-07-15 13:29:40.251463206 +0000 UTC m=+21.769087405" Jul 15 13:29:40.267547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.267504 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-spfvj" podStartSLOduration=2.736883592 podStartE2EDuration="21.267488622s" totalImagesPullingTime="18.53060503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:40.26671689 +0000 UTC m=+21.784341089" watchObservedRunningTime="2026-07-15 13:29:40.267488622 +0000 UTC m=+21.785112821" Jul 15 13:29:40.282198 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:40.282161 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hkm5q" podStartSLOduration=2.75605043 podStartE2EDuration="21.282149299s" totalImagesPullingTime="18.526098869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:40.281789894 +0000 UTC m=+21.799414099" watchObservedRunningTime="2026-07-15 13:29:40.282149299 +0000 UTC m=+21.799773498" Jul 15 13:29:41.090922 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.090885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:41.090922 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.090909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:41.091099 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:41.091027 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:41.091174 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:41.091157 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:41.184625 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.184595 2576 generic.go:356] "Generic (PLEG): container finished" podID="03c7b4c8-4578-4224-b998-2c729967387f" containerID="084d6e8f84870c3ab05ba306ae322f1f4c08c7a51f2b82b55b05782a02de3c53" exitCode=0 Jul 15 13:29:41.185145 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.184679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerDied","Data":"084d6e8f84870c3ab05ba306ae322f1f4c08c7a51f2b82b55b05782a02de3c53"} Jul 15 13:29:41.186207 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.186188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw8kv" event={"ID":"70669082-cd5b-4f86-9466-9a95cc20a578","Type":"ContainerStarted","Data":"321b82cd68ea34b2e80441beb6de91c50f986cf2a88dd7818c9081dbb8f89490"} Jul 15 13:29:41.189804 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.189780 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:29:41.190173 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.190145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"0b5a23e95732556b417fe61f4e9f897680fd984308b2fdfaef0e51f1cfd4e041"} Jul 15 13:29:41.190258 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.190217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"31e8e3b9d5620284dcd7ad3cd997951cd0cabe3ccf72b67063ee1a5ae6bb87a3"} Jul 15 13:29:41.190319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.190267 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"04a6d6a95f8050158962ff2bb3e3a5ad9bef5f2b2b7f7b8fdd3f52474ab6e204"} Jul 15 13:29:41.190319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.190313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"9146b9c35da22cb9ea67015fc94193b1d61c2f9b298c0f454abf486f3b2b6a95"} Jul 15 13:29:41.191553 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.191519 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7r9tz" event={"ID":"b16723e3-98fe-44ca-b9c4-c33598b42571","Type":"ContainerStarted","Data":"1220a83c6e5152883be25e61fe594f31fc033699dbc41611fb2160f3f84f6ce7"} Jul 15 13:29:41.224569 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.224547 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 13:29:41.228582 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.228543 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bw8kv" podStartSLOduration=3.244221006 podStartE2EDuration="22.22853009s" totalImagesPullingTime="18.984309084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:41.228193581 +0000 UTC m=+22.745817809" watchObservedRunningTime="2026-07-15 13:29:41.22853009 +0000 UTC m=+22.746154287" Jul 15 13:29:41.242121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.242086 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7r9tz" podStartSLOduration=3.6390530720000003 podStartE2EDuration="22.242072854s" totalImagesPullingTime="18.603019782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:41.241802494 +0000 UTC m=+22.759426694" watchObservedRunningTime="2026-07-15 13:29:41.242072854 +0000 UTC m=+22.759697053" Jul 15 13:29:41.919967 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.919851 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T13:29:41.224563599Z","UUID":"1a2a35d1-5990-44c4-9922-c26650b9c672","Handler":null,"Name":"","Endpoint":""} Jul 15 13:29:41.922059 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.922035 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 13:29:41.922187 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:41.922080 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 13:29:42.090118 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:42.090078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:42.090355 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:42.090244 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:42.197088 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:42.196896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" event={"ID":"7bf07266-916e-4589-a369-a353d25ec86b","Type":"ContainerStarted","Data":"1a56d32a93052d7d683f242ee2857045fe6847b455a1d7294e84e12459001b91"} Jul 15 13:29:43.089884 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.089851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:43.090066 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:43.089986 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:43.090066 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.090039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:43.090184 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:43.090165 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:43.201975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.201931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" event={"ID":"7bf07266-916e-4589-a369-a353d25ec86b","Type":"ContainerStarted","Data":"9e16e7f949f09cdbc93fa97cc4bb830fb699dedb36f935978981d88c36a84620"} Jul 15 13:29:43.205537 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.205516 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:29:43.205942 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.205914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"109bd8414561d1ef8a5f9b5ea721ffc6a6d4abc9c38de25d92d800526dadc6bf"} Jul 15 13:29:43.225738 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.225698 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pnzs7" podStartSLOduration=3.846221489 podStartE2EDuration="24.225684145s" totalImagesPullingTime="20.379462656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:43.225121699 +0000 UTC m=+24.742745898" watchObservedRunningTime="2026-07-15 13:29:43.225684145 +0000 UTC m=+24.743308343" Jul 15 13:29:43.864532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.864502 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:43.865239 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:43.865215 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:44.090524 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:44.090495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:44.090696 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:44.090662 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:44.208250 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:44.208134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:44.208773 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:44.208488 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q657c" Jul 15 13:29:45.090857 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.090834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:45.090957 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.090841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:45.090996 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:45.090966 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:45.091038 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:45.091004 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:45.214065 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.214025 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:29:45.214730 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.214388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"2004d177d0dc8115d764dd3a4680274ac413212abcdf455ddfe6c6b31b3e5107"} Jul 15 13:29:45.215881 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.214822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:45.215881 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.214886 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:45.215881 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.215027 2576 scope.go:117] "RemoveContainer" containerID="30b5ad8c30a246868c66a0273b6ef16b86024a99abecd105644f2425e782a556" Jul 15 13:29:45.231332 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.231217 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:45.231439 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:45.231426 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:46.089861 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:46.089840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:46.090000 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:46.089928 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:46.217647 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:46.217611 2576 generic.go:356] "Generic (PLEG): container finished" podID="03c7b4c8-4578-4224-b998-2c729967387f" containerID="f3171d774dfe2aa42db1f4df6719e42ad6a31b1c296753a0273b3fa2d2af831f" exitCode=0 Jul 15 13:29:46.218026 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:46.217694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerDied","Data":"f3171d774dfe2aa42db1f4df6719e42ad6a31b1c296753a0273b3fa2d2af831f"} Jul 15 13:29:46.221465 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:46.221448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:29:46.221820 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:46.221789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" event={"ID":"944ce9ec-3ae4-431a-8f68-c6c720fc7613","Type":"ContainerStarted","Data":"07c65701d2dde3e6b7931bea78fcd202a28375b21642b4dbde972babc3192250"} Jul 15 13:29:46.221905 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:46.221865 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 15 13:29:46.274215 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:46.274168 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" podStartSLOduration=8.594081521 podStartE2EDuration="27.274152685s" totalImagesPullingTime="18.680071164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:46.273849401 +0000 UTC m=+27.791473599" watchObservedRunningTime="2026-07-15 13:29:46.274152685 +0000 UTC m=+27.791776883" Jul 15 13:29:47.090953 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.090873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:47.091117 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.090873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:47.091117 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:47.091031 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:47.091222 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:47.091111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:47.223986 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.223964 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 15 13:29:47.356664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.356603 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:29:47.435102 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.434775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tkwn8"] Jul 15 13:29:47.435102 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.435082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:47.435247 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:47.435180 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:47.435883 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.435859 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b62v4"] Jul 15 13:29:47.435982 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.435957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:47.436086 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:47.436064 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:47.436861 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.436836 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbsfv"] Jul 15 13:29:47.436983 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:47.436925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:47.437042 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:47.437011 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:48.227241 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:48.227211 2576 generic.go:356] "Generic (PLEG): container finished" podID="03c7b4c8-4578-4224-b998-2c729967387f" containerID="c098b55b15d4d3c5ae83245ad3d90ded01d3e16c57860ace31b9d3da52089c34" exitCode=0 Jul 15 13:29:48.227637 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:48.227291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerDied","Data":"c098b55b15d4d3c5ae83245ad3d90ded01d3e16c57860ace31b9d3da52089c34"} Jul 15 13:29:49.092115 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:49.092086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:49.092255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:49.092172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:49.092255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:49.092213 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:49.092255 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:49.092211 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:49.092421 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:49.092286 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:49.092421 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:49.092377 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:49.232303 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:49.232268 2576 generic.go:356] "Generic (PLEG): container finished" podID="03c7b4c8-4578-4224-b998-2c729967387f" containerID="e7e4f5d297ca11fc0886c387520d6e34c724a28d4346db469b95c70f5c5ee1e6" exitCode=0 Jul 15 13:29:49.232860 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:49.232310 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerDied","Data":"e7e4f5d297ca11fc0886c387520d6e34c724a28d4346db469b95c70f5c5ee1e6"} Jul 15 13:29:49.392609 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:49.392587 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spfvj_77907d2e-8cca-44e7-a03b-8b34bf3bd2bf/dns-node-resolver/0.log" Jul 15 13:29:50.169575 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:50.169547 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkm5q_d29ac1b1-707e-4791-97f3-62acd643c83d/node-ca/0.log" Jul 15 13:29:51.090723 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:51.090683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:51.090723 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:51.090703 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:51.091344 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:51.090816 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:51.091344 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:51.090891 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:51.091344 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:51.090944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:51.091344 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:51.091006 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:52.600062 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:52.600029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:52.600617 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.600208 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:52.600617 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.600302 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret podName:eafabcae-943c-4420-b244-a96aa27bedb9 nodeName:}" failed. No retries permitted until 2026-07-15 13:30:24.600280964 +0000 UTC m=+66.117905159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret") pod "global-pull-secret-syncer-tkwn8" (UID: "eafabcae-943c-4420-b244-a96aa27bedb9") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:29:52.700714 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:52.700685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:52.700875 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.700848 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:52.700944 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.700921 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs podName:62441196-0328-49dc-aeb8-714c0357b917 nodeName:}" failed. No retries permitted until 2026-07-15 13:30:24.700901366 +0000 UTC m=+66.218525559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs") pod "network-metrics-daemon-b62v4" (UID: "62441196-0328-49dc-aeb8-714c0357b917") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:29:52.802061 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:52.802023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:52.802243 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.802191 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:29:52.802243 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.802220 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:29:52.802243 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.802231 2576 projected.go:196] Error preparing data for projected volume kube-api-access-hglq2 for pod openshift-network-diagnostics/network-check-target-rbsfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:52.802378 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:52.802289 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2 podName:0609e1f4-a178-4ed3-aa1c-22c27d375e87 nodeName:}" failed. No retries permitted until 2026-07-15 13:30:24.802271729 +0000 UTC m=+66.319895908 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hglq2" (UniqueName: "kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2") pod "network-check-target-rbsfv" (UID: "0609e1f4-a178-4ed3-aa1c-22c27d375e87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:29:53.090365 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:53.090291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:53.090544 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:53.090295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:53.090544 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:53.090436 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:53.090544 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:53.090452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:53.090695 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:53.090567 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:53.090695 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:53.090649 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:55.090612 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:55.090574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:55.090975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:55.090574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:55.090975 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:55.090684 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:55.090975 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:55.090781 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:55.090975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:55.090801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:55.090975 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:55.090857 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:55.248006 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:55.247977 2576 generic.go:356] "Generic (PLEG): container finished" podID="03c7b4c8-4578-4224-b998-2c729967387f" containerID="bf840e18430e18700656c48fe9b2e77bd17a06eccde28a9b5f678492884f8ba4" exitCode=0 Jul 15 13:29:55.248156 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:55.248027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerDied","Data":"bf840e18430e18700656c48fe9b2e77bd17a06eccde28a9b5f678492884f8ba4"} Jul 15 13:29:56.252900 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:56.252703 2576 generic.go:356] "Generic (PLEG): container finished" podID="03c7b4c8-4578-4224-b998-2c729967387f" containerID="d341779c2c2d84b82576829a36b2df54452b0f9f2e3f02f9a0eb25cd1ad3a26d" exitCode=0 Jul 15 13:29:56.253242 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:56.252768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerDied","Data":"d341779c2c2d84b82576829a36b2df54452b0f9f2e3f02f9a0eb25cd1ad3a26d"} Jul 15 13:29:57.090515 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:57.090443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:57.090649 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:57.090443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:57.090649 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:57.090447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:57.090714 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:57.090529 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:57.090714 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:57.090620 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:29:57.090714 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:57.090702 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:57.257994 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:57.257958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6mprk" event={"ID":"03c7b4c8-4578-4224-b998-2c729967387f","Type":"ContainerStarted","Data":"d4ad94cfb0c8686dd255ecf09dca0bfb14c825786d8b666148ad29b582cfdadc"} Jul 15 13:29:57.284044 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:57.283981 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6mprk" podStartSLOduration=8.677299879 podStartE2EDuration="38.283965454s" totalImagesPullingTime="29.606665575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:29:57.283667354 +0000 UTC m=+38.801291554" watchObservedRunningTime="2026-07-15 13:29:57.283965454 +0000 UTC m=+38.801589653" Jul 15 13:29:59.090860 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:59.090819 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:29:59.091281 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:59.090918 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:29:59.091281 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:59.091003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:29:59.091281 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:29:59.091096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:29:59.091281 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:59.091115 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:29:59.091281 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:29:59.091156 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:01.090054 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:01.090023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:01.090540 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:01.090025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:01.090540 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:01.090139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:01.090540 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:01.090225 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:30:01.090540 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:01.090035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:01.090540 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:01.090328 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:30:03.090296 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:03.090259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:03.090732 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:03.090260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:03.090732 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:03.090364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:30:03.090732 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:03.090266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:03.090732 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:03.090455 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:30:03.090732 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:03.090542 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:05.090010 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:05.089982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:05.090010 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:05.090004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:05.090420 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:05.089982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:05.090420 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:05.090086 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:05.090420 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:05.090156 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:30:05.090420 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:05.090203 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:30:07.090154 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:07.090113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:07.090578 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:07.090224 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:30:07.090578 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:07.090260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:07.090578 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:07.090113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:07.090578 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:07.090356 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:30:07.090578 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:07.090461 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:09.091479 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:09.091453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:09.091911 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:09.091530 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:30:09.091911 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:09.091614 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:09.091911 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:09.091696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:09.091911 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:09.091722 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:09.091911 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:09.091746 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:30:11.090893 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:11.090866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:11.090893 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:11.090882 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:11.091287 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:11.090959 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:30:11.091287 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:11.091017 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:11.091287 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:11.091060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:11.091287 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:11.091138 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:30:13.090089 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.090060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:13.090089 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.090079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:13.090604 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:13.090168 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tkwn8" podUID="eafabcae-943c-4420-b244-a96aa27bedb9" Jul 15 13:30:13.090604 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:13.090224 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b62v4" podUID="62441196-0328-49dc-aeb8-714c0357b917" Jul 15 13:30:13.090604 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.090266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:13.090604 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:13.090344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbsfv" podUID="0609e1f4-a178-4ed3-aa1c-22c27d375e87" Jul 15 13:30:13.812664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.812638 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeReady" Jul 15 13:30:13.812812 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.812757 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 13:30:13.898870 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.898799 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dgm75"] Jul 15 13:30:13.900571 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.900555 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z784n"] Jul 15 13:30:13.900696 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.900682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:13.902447 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.902428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:13.904884 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.904869 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 13:30:13.905013 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.904996 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rnzhl\"" Jul 15 13:30:13.905370 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.905356 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 13:30:13.905572 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.905555 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 13:30:13.905749 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.905586 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 13:30:13.905749 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.905604 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 13:30:13.906050 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.906035 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vgfvj\"" Jul 15 13:30:13.906464 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.906448 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 13:30:13.922849 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.922822 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z784n"] Jul 15 13:30:13.923331 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.923313 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dgm75"] Jul 15 13:30:13.928312 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.928296 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5gfdp"] Jul 15 13:30:13.929998 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.929982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:13.932872 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.932852 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 13:30:13.933088 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.933075 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qvx6p\"" Jul 15 13:30:13.933176 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.933123 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 13:30:13.933455 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.933441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 13:30:13.945025 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.945006 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gfdp"] Jul 15 13:30:13.958511 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chx5\" (UniqueName: \"kubernetes.io/projected/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-kube-api-access-5chx5\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:13.958597 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-tmp-dir\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:13.958597 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-metrics-tls\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:13.958597 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrkv\" (UniqueName: \"kubernetes.io/projected/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-kube-api-access-ztrkv\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:13.958753 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-crio-socket\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:13.958753 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-data-volume\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:13.958753 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-config-volume\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:13.958753 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:13.958893 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:13.958776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059209 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-data-volume\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059209 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-config-volume\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.059383 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059443 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059443 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb60e8c3-59de-4093-9c04-d6904b54625f-cert\") pod \"ingress-canary-5gfdp\" (UID: \"eb60e8c3-59de-4093-9c04-d6904b54625f\") " pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:14.059532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/eb60e8c3-59de-4093-9c04-d6904b54625f-kube-api-access-ghnqm\") pod \"ingress-canary-5gfdp\" (UID: \"eb60e8c3-59de-4093-9c04-d6904b54625f\") " pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:14.059532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chx5\" (UniqueName: \"kubernetes.io/projected/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-kube-api-access-5chx5\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.059627 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-tmp-dir\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.059627 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-metrics-tls\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.059725 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrkv\" (UniqueName: \"kubernetes.io/projected/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-kube-api-access-ztrkv\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059725 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-crio-socket\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059725 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-data-volume\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059889 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-crio-socket\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059889 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.059972 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059886 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-config-volume\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.059972 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.059871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-tmp-dir\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.063314 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.063287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-metrics-tls\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.063424 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.063287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.077648 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.077623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrkv\" (UniqueName: \"kubernetes.io/projected/27ca2901-5e9e-42a7-ad5f-1b586f3667bf-kube-api-access-ztrkv\") pod \"insights-runtime-extractor-z784n\" (UID: \"27ca2901-5e9e-42a7-ad5f-1b586f3667bf\") " pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.078186 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.078156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chx5\" (UniqueName: \"kubernetes.io/projected/8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6-kube-api-access-5chx5\") pod \"dns-default-dgm75\" (UID: \"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6\") " pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.160581 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.160519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb60e8c3-59de-4093-9c04-d6904b54625f-cert\") pod \"ingress-canary-5gfdp\" (UID: \"eb60e8c3-59de-4093-9c04-d6904b54625f\") " pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:14.160581 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.160547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/eb60e8c3-59de-4093-9c04-d6904b54625f-kube-api-access-ghnqm\") pod \"ingress-canary-5gfdp\" (UID: \"eb60e8c3-59de-4093-9c04-d6904b54625f\") " pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:14.162667 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.162648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb60e8c3-59de-4093-9c04-d6904b54625f-cert\") pod \"ingress-canary-5gfdp\" (UID: \"eb60e8c3-59de-4093-9c04-d6904b54625f\") " pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:14.187722 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.187698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnqm\" (UniqueName: \"kubernetes.io/projected/eb60e8c3-59de-4093-9c04-d6904b54625f-kube-api-access-ghnqm\") pod \"ingress-canary-5gfdp\" (UID: \"eb60e8c3-59de-4093-9c04-d6904b54625f\") " pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:14.211143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.211122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:14.217272 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.217257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z784n" Jul 15 13:30:14.238299 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.238272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gfdp" Jul 15 13:30:14.379818 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.379509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z784n"] Jul 15 13:30:14.402926 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.402783 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dgm75"] Jul 15 13:30:14.403665 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:14.403641 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gfdp"] Jul 15 13:30:14.406974 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:14.406950 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb60e8c3_59de_4093_9c04_d6904b54625f.slice/crio-78d628d9f576a253690ab22e380c266a2c398ebf7ce83f91e8dbe21ff573fe05 WatchSource:0}: Error finding container 78d628d9f576a253690ab22e380c266a2c398ebf7ce83f91e8dbe21ff573fe05: Status 404 returned error can't find the container with id 78d628d9f576a253690ab22e380c266a2c398ebf7ce83f91e8dbe21ff573fe05 Jul 15 13:30:14.407628 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:14.407554 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb3cce7_d6f7_4803_99fc_9fa7ff79a8f6.slice/crio-5ba653af9d0e3147013db2d4770461c193b955d1e3341298f56ef69ca81f8f51 WatchSource:0}: Error finding container 5ba653af9d0e3147013db2d4770461c193b955d1e3341298f56ef69ca81f8f51: Status 404 returned error can't find the container with id 5ba653af9d0e3147013db2d4770461c193b955d1e3341298f56ef69ca81f8f51 Jul 15 13:30:15.091534 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.091504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:15.092378 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.092340 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:15.092733 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.092701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:15.096563 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.096540 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 13:30:15.096843 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.096802 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rv6f8\"" Jul 15 13:30:15.097092 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.097066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 13:30:15.097186 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.097074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 13:30:15.097438 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.097422 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w89gv\"" Jul 15 13:30:15.098491 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.098473 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 13:30:15.301600 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.301511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dgm75" event={"ID":"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6","Type":"ContainerStarted","Data":"5ba653af9d0e3147013db2d4770461c193b955d1e3341298f56ef69ca81f8f51"} Jul 15 13:30:15.302944 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.302861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gfdp" event={"ID":"eb60e8c3-59de-4093-9c04-d6904b54625f","Type":"ContainerStarted","Data":"78d628d9f576a253690ab22e380c266a2c398ebf7ce83f91e8dbe21ff573fe05"} Jul 15 13:30:15.304566 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.304485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z784n" event={"ID":"27ca2901-5e9e-42a7-ad5f-1b586f3667bf","Type":"ContainerStarted","Data":"ce953966c98af2ef85d13b5564f1fc599516f6583f12039e3e480eaa481b2015"} Jul 15 13:30:15.731566 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.731195 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86799ff95b-mnbb5"] Jul 15 13:30:15.742515 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.742488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.748740 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.748645 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 13:30:15.749975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.748901 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 13:30:15.749975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.749127 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-89mch\"" Jul 15 13:30:15.749975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.749330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 13:30:15.749975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.749751 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 13:30:15.749975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.749801 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 13:30:15.749975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.749974 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 13:30:15.760664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.760636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86799ff95b-mnbb5"] Jul 15 13:30:15.765391 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.765360 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 13:30:15.765809 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.765791 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 13:30:15.885673 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.885625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-serving-cert\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.885858 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.885688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-config\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.885858 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.885716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhv6\" (UniqueName: \"kubernetes.io/projected/70479f4d-58a4-45b4-8870-0e84b7d4eec2-kube-api-access-fdhv6\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.885977 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.885902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-oauth-config\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.885977 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.885938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-service-ca\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.885977 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.885963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-oauth-serving-cert\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.886119 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.885994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-trusted-ca-bundle\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.986391 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.986306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-oauth-config\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.986391 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.986357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-service-ca\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.986391 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.986380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-oauth-serving-cert\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.986660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.986425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-trusted-ca-bundle\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.986660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.986453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-serving-cert\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.986660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.986598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-config\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.986660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.986623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhv6\" (UniqueName: \"kubernetes.io/projected/70479f4d-58a4-45b4-8870-0e84b7d4eec2-kube-api-access-fdhv6\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.987274 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.987248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-service-ca\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.987691 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.987669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-oauth-serving-cert\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.988249 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.988222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-config\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.988445 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.988424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-trusted-ca-bundle\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.990991 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.990968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-oauth-config\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:15.998137 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:15.998106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-serving-cert\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:16.010307 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:16.010261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhv6\" (UniqueName: \"kubernetes.io/projected/70479f4d-58a4-45b4-8870-0e84b7d4eec2-kube-api-access-fdhv6\") pod \"console-86799ff95b-mnbb5\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:16.053581 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:16.053546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:16.660465 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:16.660326 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86799ff95b-mnbb5"] Jul 15 13:30:16.774792 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:16.774704 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70479f4d_58a4_45b4_8870_0e84b7d4eec2.slice/crio-32e4f17e503d89caaabda5fadef036513891df66e454b6bd852c6f136ceb7d56 WatchSource:0}: Error finding container 32e4f17e503d89caaabda5fadef036513891df66e454b6bd852c6f136ceb7d56: Status 404 returned error can't find the container with id 32e4f17e503d89caaabda5fadef036513891df66e454b6bd852c6f136ceb7d56 Jul 15 13:30:16.998948 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:16.998913 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd"] Jul 15 13:30:17.007295 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.007271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" Jul 15 13:30:17.011184 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.011155 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nchnf\"" Jul 15 13:30:17.011752 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.011579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 15 13:30:17.028526 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.028475 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd"] Jul 15 13:30:17.097167 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.097134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40166e11-b01b-449c-9126-30bac206854e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bm9dd\" (UID: \"40166e11-b01b-449c-9126-30bac206854e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" Jul 15 13:30:17.197487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.197452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40166e11-b01b-449c-9126-30bac206854e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bm9dd\" (UID: \"40166e11-b01b-449c-9126-30bac206854e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" Jul 15 13:30:17.200184 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.200155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40166e11-b01b-449c-9126-30bac206854e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-bm9dd\" (UID: \"40166e11-b01b-449c-9126-30bac206854e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" Jul 15 13:30:17.311594 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.311509 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86799ff95b-mnbb5" event={"ID":"70479f4d-58a4-45b4-8870-0e84b7d4eec2","Type":"ContainerStarted","Data":"32e4f17e503d89caaabda5fadef036513891df66e454b6bd852c6f136ceb7d56"} Jul 15 13:30:17.317784 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:17.317760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" Jul 15 13:30:18.238065 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:18.238043 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rps9b" Jul 15 13:30:20.323770 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:20.323727 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dgm75" event={"ID":"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6","Type":"ContainerStarted","Data":"0f74097e318e507c4cefca0233501264e654c0c0b34535d1a184be777995e52d"} Jul 15 13:30:20.331774 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:20.331692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z784n" event={"ID":"27ca2901-5e9e-42a7-ad5f-1b586f3667bf","Type":"ContainerStarted","Data":"295aaf3f86162e21eb3d45787be6646b1a0653173149280e66f85d6f079e719d"} Jul 15 13:30:20.335489 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:20.334869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gfdp" event={"ID":"eb60e8c3-59de-4093-9c04-d6904b54625f","Type":"ContainerStarted","Data":"ded9d712c5d7cf22d3dded698b935837097c2ef5b7c1983972d6348bf1a696ef"} Jul 15 13:30:20.345980 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:20.345676 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd"] Jul 15 13:30:20.362188 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:20.362155 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40166e11_b01b_449c_9126_30bac206854e.slice/crio-ab439b1f8a1c3da31842890fc3aebb36f279878818a73a3454a3e74e20a226af WatchSource:0}: Error finding container ab439b1f8a1c3da31842890fc3aebb36f279878818a73a3454a3e74e20a226af: Status 404 returned error can't find the container with id ab439b1f8a1c3da31842890fc3aebb36f279878818a73a3454a3e74e20a226af Jul 15 13:30:20.372712 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:20.372671 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5gfdp" podStartSLOduration=1.629856512 podStartE2EDuration="7.372650304s" totalImagesPullingTime="5.742793792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:20.372575612 +0000 UTC m=+61.890199811" watchObservedRunningTime="2026-07-15 13:30:20.372650304 +0000 UTC m=+61.890274502" Jul 15 13:30:21.343652 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:21.341093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" event={"ID":"40166e11-b01b-449c-9126-30bac206854e","Type":"ContainerStarted","Data":"ab439b1f8a1c3da31842890fc3aebb36f279878818a73a3454a3e74e20a226af"} Jul 15 13:30:21.345647 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:21.345562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dgm75" event={"ID":"8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6","Type":"ContainerStarted","Data":"aaa30d85cf2c432e904e59e504d3413b33e9b881a298d0ab931ff8ed539ffb44"} Jul 15 13:30:21.345807 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:21.345793 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:21.382669 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:21.382613 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dgm75" podStartSLOduration=3.019242528 podStartE2EDuration="8.382594615s" totalImagesPullingTime="5.363352087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:21.378748236 +0000 UTC m=+62.896372461" watchObservedRunningTime="2026-07-15 13:30:21.382594615 +0000 UTC m=+62.900218815" Jul 15 13:30:24.357034 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.356999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z784n" event={"ID":"27ca2901-5e9e-42a7-ad5f-1b586f3667bf","Type":"ContainerStarted","Data":"08cc2fa52ca110a963c5a8c183196ace5c89b4f9b018508af2bc53c328d5ab9c"} Jul 15 13:30:24.358501 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.358476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86799ff95b-mnbb5" event={"ID":"70479f4d-58a4-45b4-8870-0e84b7d4eec2","Type":"ContainerStarted","Data":"7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b"} Jul 15 13:30:24.359807 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.359786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" event={"ID":"40166e11-b01b-449c-9126-30bac206854e","Type":"ContainerStarted","Data":"f15c446998b29ccc467457d4cb28b9974d3416d3d22f97ff708cd12b9cd698bb"} Jul 15 13:30:24.359994 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.359979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" Jul 15 13:30:24.364071 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.364052 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" Jul 15 13:30:24.422927 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.422835 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86799ff95b-mnbb5" podStartSLOduration=2.086745599 podStartE2EDuration="9.422820127s" totalImagesPullingTime="7.336074528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:24.42229329 +0000 UTC m=+65.939917490" watchObservedRunningTime="2026-07-15 13:30:24.422820127 +0000 UTC m=+65.940444398" Jul 15 13:30:24.454692 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.454652 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-bm9dd" podStartSLOduration=4.628897491 podStartE2EDuration="8.454639423s" totalImagesPullingTime="3.825741932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:24.453982864 +0000 UTC m=+65.971607063" watchObservedRunningTime="2026-07-15 13:30:24.454639423 +0000 UTC m=+65.972263621" Jul 15 13:30:24.661411 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.661372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:24.668388 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.668368 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 13:30:24.673767 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.673712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eafabcae-943c-4420-b244-a96aa27bedb9-original-pull-secret\") pod \"global-pull-secret-syncer-tkwn8\" (UID: \"eafabcae-943c-4420-b244-a96aa27bedb9\") " pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:24.717000 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.716978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tkwn8" Jul 15 13:30:24.762628 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.762594 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:24.767661 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.767641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 13:30:24.775735 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.775709 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62441196-0328-49dc-aeb8-714c0357b917-metrics-certs\") pod \"network-metrics-daemon-b62v4\" (UID: \"62441196-0328-49dc-aeb8-714c0357b917\") " pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:24.847708 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.847675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tkwn8"] Jul 15 13:30:24.852043 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:24.852012 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafabcae_943c_4420_b244_a96aa27bedb9.slice/crio-bf4680484edb0f40134bec30728a5bb717aca1e6915316bc5dd7c56b91976578 WatchSource:0}: Error finding container bf4680484edb0f40134bec30728a5bb717aca1e6915316bc5dd7c56b91976578: Status 404 returned error can't find the container with id bf4680484edb0f40134bec30728a5bb717aca1e6915316bc5dd7c56b91976578 Jul 15 13:30:24.863857 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.863829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:24.867599 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.867580 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 13:30:24.876877 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.876857 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 13:30:24.886991 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:24.886964 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglq2\" (UniqueName: \"kubernetes.io/projected/0609e1f4-a178-4ed3-aa1c-22c27d375e87-kube-api-access-hglq2\") pod \"network-check-target-rbsfv\" (UID: \"0609e1f4-a178-4ed3-aa1c-22c27d375e87\") " pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:25.012792 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.012721 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rv6f8\"" Jul 15 13:30:25.018031 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.017983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:25.028486 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.028464 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w89gv\"" Jul 15 13:30:25.035419 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.035384 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b62v4" Jul 15 13:30:25.111692 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.111154 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-cjp8v"] Jul 15 13:30:25.121923 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.120627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.125581 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.125472 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jnjq4\"" Jul 15 13:30:25.126896 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.125886 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 13:30:25.126896 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.126089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 15 13:30:25.126896 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.126539 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 13:30:25.126896 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.126881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 13:30:25.127461 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.127069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 15 13:30:25.147101 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.147004 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-cjp8v"] Jul 15 13:30:25.224565 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.224495 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbsfv"] Jul 15 13:30:25.228959 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:25.228893 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0609e1f4_a178_4ed3_aa1c_22c27d375e87.slice/crio-f1406262ea4a74b29a2a63b52def173cdb366df66ccae5e45d5b2c5170394b14 WatchSource:0}: Error finding container f1406262ea4a74b29a2a63b52def173cdb366df66ccae5e45d5b2c5170394b14: Status 404 returned error can't find the container with id f1406262ea4a74b29a2a63b52def173cdb366df66ccae5e45d5b2c5170394b14 Jul 15 13:30:25.237097 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.237075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b62v4"] Jul 15 13:30:25.240977 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:25.240951 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62441196_0328_49dc_aeb8_714c0357b917.slice/crio-7e68b17201b3627ede85b9c59ad2b3582ed7357e599438f58c3ca0ad31a8ad46 WatchSource:0}: Error finding container 7e68b17201b3627ede85b9c59ad2b3582ed7357e599438f58c3ca0ad31a8ad46: Status 404 returned error can't find the container with id 7e68b17201b3627ede85b9c59ad2b3582ed7357e599438f58c3ca0ad31a8ad46 Jul 15 13:30:25.266344 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.266284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2623a670-f328-48f5-9e06-d77bb36bc903-metrics-client-ca\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.266482 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.266344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2623a670-f328-48f5-9e06-d77bb36bc903-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.266482 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.266375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hgc\" (UniqueName: \"kubernetes.io/projected/2623a670-f328-48f5-9e06-d77bb36bc903-kube-api-access-f9hgc\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.266592 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.266477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2623a670-f328-48f5-9e06-d77bb36bc903-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.364472 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.364424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tkwn8" event={"ID":"eafabcae-943c-4420-b244-a96aa27bedb9","Type":"ContainerStarted","Data":"bf4680484edb0f40134bec30728a5bb717aca1e6915316bc5dd7c56b91976578"} Jul 15 13:30:25.366048 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.366010 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b62v4" event={"ID":"62441196-0328-49dc-aeb8-714c0357b917","Type":"ContainerStarted","Data":"7e68b17201b3627ede85b9c59ad2b3582ed7357e599438f58c3ca0ad31a8ad46"} Jul 15 13:30:25.366982 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.366753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2623a670-f328-48f5-9e06-d77bb36bc903-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.366982 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.366787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hgc\" (UniqueName: \"kubernetes.io/projected/2623a670-f328-48f5-9e06-d77bb36bc903-kube-api-access-f9hgc\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.366982 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.366844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2623a670-f328-48f5-9e06-d77bb36bc903-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.366982 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.366876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2623a670-f328-48f5-9e06-d77bb36bc903-metrics-client-ca\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.367547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.367419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbsfv" event={"ID":"0609e1f4-a178-4ed3-aa1c-22c27d375e87","Type":"ContainerStarted","Data":"f1406262ea4a74b29a2a63b52def173cdb366df66ccae5e45d5b2c5170394b14"} Jul 15 13:30:25.367735 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.367688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2623a670-f328-48f5-9e06-d77bb36bc903-metrics-client-ca\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.369784 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.369750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2623a670-f328-48f5-9e06-d77bb36bc903-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.369892 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.369760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2623a670-f328-48f5-9e06-d77bb36bc903-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.377673 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.377631 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hgc\" (UniqueName: \"kubernetes.io/projected/2623a670-f328-48f5-9e06-d77bb36bc903-kube-api-access-f9hgc\") pod \"prometheus-operator-64785b6675-cjp8v\" (UID: \"2623a670-f328-48f5-9e06-d77bb36bc903\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.434503 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.434483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" Jul 15 13:30:25.587379 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:25.587343 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-cjp8v"] Jul 15 13:30:25.589959 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:25.589929 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2623a670_f328_48f5_9e06_d77bb36bc903.slice/crio-a01e2bcb1aae09440c383fcf77f0c840f1e081dc9b4dd2ef6ea9d44938e1a2fa WatchSource:0}: Error finding container a01e2bcb1aae09440c383fcf77f0c840f1e081dc9b4dd2ef6ea9d44938e1a2fa: Status 404 returned error can't find the container with id a01e2bcb1aae09440c383fcf77f0c840f1e081dc9b4dd2ef6ea9d44938e1a2fa Jul 15 13:30:26.054792 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:26.054758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:26.054951 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:26.054852 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:26.055938 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:26.055905 2576 patch_prober.go:28] interesting pod/console-86799ff95b-mnbb5 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 15 13:30:26.056060 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:26.055960 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-86799ff95b-mnbb5" podUID="70479f4d-58a4-45b4-8870-0e84b7d4eec2" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 15 13:30:26.376623 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:26.376585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" event={"ID":"2623a670-f328-48f5-9e06-d77bb36bc903","Type":"ContainerStarted","Data":"a01e2bcb1aae09440c383fcf77f0c840f1e081dc9b4dd2ef6ea9d44938e1a2fa"} Jul 15 13:30:31.351279 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.350793 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dgm75" Jul 15 13:30:31.401312 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.401276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z784n" event={"ID":"27ca2901-5e9e-42a7-ad5f-1b586f3667bf","Type":"ContainerStarted","Data":"02e627aa65440de6324c155e2599b4145b84295eea18d1423d240a84d9b86e83"} Jul 15 13:30:31.404267 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.404049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b62v4" event={"ID":"62441196-0328-49dc-aeb8-714c0357b917","Type":"ContainerStarted","Data":"8113da71e3da0a92a8333511058a315c00dcd0db87df94e64bcad756c4793ff3"} Jul 15 13:30:31.404267 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.404113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b62v4" event={"ID":"62441196-0328-49dc-aeb8-714c0357b917","Type":"ContainerStarted","Data":"7c9030cb89bfdf355d3cc9456a07b0dc07e71cb6a91930c8c9b2f0ec168c3d35"} Jul 15 13:30:31.406507 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.406174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbsfv" event={"ID":"0609e1f4-a178-4ed3-aa1c-22c27d375e87","Type":"ContainerStarted","Data":"360baae1e26fcd55e585940cc49b6b783df75bb24eb28376a2dd0eaf477092bb"} Jul 15 13:30:31.406507 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.406318 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:30:31.408592 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.408572 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" event={"ID":"2623a670-f328-48f5-9e06-d77bb36bc903","Type":"ContainerStarted","Data":"788ce51c9c7efece306ff85cae0250190983330d1872693fbf73e0914fd820c7"} Jul 15 13:30:31.408697 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.408620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" event={"ID":"2623a670-f328-48f5-9e06-d77bb36bc903","Type":"ContainerStarted","Data":"12bfe5b61b7761d05367a6792e8465fdcfd87ec6648bcd5c8f9004f69884663b"} Jul 15 13:30:31.426305 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.425761 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z784n" podStartSLOduration=2.5734530429999998 podStartE2EDuration="18.425744661s" totalImagesPullingTime="15.852291618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:30:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:31.424780394 +0000 UTC m=+72.942404593" watchObservedRunningTime="2026-07-15 13:30:31.425744661 +0000 UTC m=+72.943368860" Jul 15 13:30:31.454499 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.454457 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-cjp8v" podStartSLOduration=1.647316438 podStartE2EDuration="6.454443499s" totalImagesPullingTime="4.807127061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:31.453297972 +0000 UTC m=+72.970922171" watchObservedRunningTime="2026-07-15 13:30:31.454443499 +0000 UTC m=+72.972067698" Jul 15 13:30:31.482872 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.482832 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b62v4" podStartSLOduration=67.325764823 podStartE2EDuration="1m12.482819104s" totalImagesPullingTime="5.157054281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:31.482617748 +0000 UTC m=+73.000241949" watchObservedRunningTime="2026-07-15 13:30:31.482819104 +0000 UTC m=+73.000443318" Jul 15 13:30:31.509026 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:31.505570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rbsfv" podStartSLOduration=67.321389221 podStartE2EDuration="1m12.505547781s" totalImagesPullingTime="5.18415856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:31.504815127 +0000 UTC m=+73.022439324" watchObservedRunningTime="2026-07-15 13:30:31.505547781 +0000 UTC m=+73.023171984" Jul 15 13:30:32.413646 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:32.413601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tkwn8" event={"ID":"eafabcae-943c-4420-b244-a96aa27bedb9","Type":"ContainerStarted","Data":"b216c4d50af4180a85202ad1efc1c896c9b28f1536d24e368be0fc5455def822"} Jul 15 13:30:32.430878 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:32.430834 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tkwn8" podStartSLOduration=66.308052762 podStartE2EDuration="1m13.430819407s" totalImagesPullingTime="7.122766645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:32.430371744 +0000 UTC m=+73.947995944" watchObservedRunningTime="2026-07-15 13:30:32.430819407 +0000 UTC m=+73.948443632" Jul 15 13:30:33.742546 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.742515 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6"] Jul 15 13:30:33.744680 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.744661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.747172 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.747154 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 13:30:33.747172 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.747161 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 13:30:33.747339 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.747320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-985xf\"" Jul 15 13:30:33.765448 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.765419 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-kg5vz"] Jul 15 13:30:33.767467 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.767446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6"] Jul 15 13:30:33.767564 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.767539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:33.769364 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.769342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 13:30:33.769481 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.769390 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r2d5s"] Jul 15 13:30:33.769536 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.769513 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 13:30:33.769820 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.769804 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p6hhr\"" Jul 15 13:30:33.770432 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.770391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 13:30:33.771602 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.771586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.773363 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.773344 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 13:30:33.773522 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.773449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jpzss\"" Jul 15 13:30:33.773595 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.773524 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 13:30:33.773814 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.773799 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 13:30:33.786277 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.786256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-kg5vz"] Jul 15 13:30:33.828994 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.828957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.828994 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.828992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.829191 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.829072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.829191 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.829134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brtgn\" (UniqueName: \"kubernetes.io/projected/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-kube-api-access-brtgn\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.929832 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.929801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:33.929832 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.929839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930023 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.929861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-textfile\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930023 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.929922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-sys\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930023 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.929953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930023 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.929986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:33.930023 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.930250 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-root\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930250 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcb8\" (UniqueName: \"kubernetes.io/projected/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-kube-api-access-lpcb8\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930250 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:33.930126 2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 15 13:30:33.930250 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brtgn\" (UniqueName: \"kubernetes.io/projected/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-kube-api-access-brtgn\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.930250 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:33.930182 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-tls podName:eda49ddc-ac2f-41ce-a01e-5c0efddf15f8 nodeName:}" failed. No retries permitted until 2026-07-15 13:30:34.430161469 +0000 UTC m=+75.947785660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-z6mn6" (UID: "eda49ddc-ac2f-41ce-a01e-5c0efddf15f8") : secret "openshift-state-metrics-tls" not found Jul 15 13:30:33.930250 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930250 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-wtmp\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.930532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.930532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:33.930532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:33.930532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-metrics-client-ca\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:33.930532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxjt\" (UniqueName: \"kubernetes.io/projected/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-api-access-kpxjt\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:33.930532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.930516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:33.931027 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.931000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.932677 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.932661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:33.940263 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:33.940242 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brtgn\" (UniqueName: \"kubernetes.io/projected/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-kube-api-access-brtgn\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:34.031547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.031547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.031547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-metrics-client-ca\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.031794 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxjt\" (UniqueName: \"kubernetes.io/projected/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-api-access-kpxjt\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.031794 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.031794 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.031794 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.031794 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-textfile\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.031794 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-sys\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.031794 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032099 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.032099 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-root\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032099 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcb8\" (UniqueName: \"kubernetes.io/projected/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-kube-api-access-lpcb8\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032099 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032099 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.031992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-wtmp\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032099 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-root\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:34.032175 2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-wtmp\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-metrics-client-ca\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:34.032247 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-tls podName:41cd44c2-5d29-46c9-ae23-b1efb285f8c7 nodeName:}" failed. No retries permitted until 2026-07-15 13:30:34.532213744 +0000 UTC m=+76.049837937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-kg5vz" (UID: "41cd44c2-5d29-46c9-ae23-b1efb285f8c7") : secret "kube-state-metrics-tls" not found Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:34.032301 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.032364 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:34.032353 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls podName:3299a86f-8ce9-442e-8c4d-1b40cf4e49a6 nodeName:}" failed. No retries permitted until 2026-07-15 13:30:34.532338536 +0000 UTC m=+76.049962715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls") pod "node-exporter-r2d5s" (UID: "3299a86f-8ce9-442e-8c4d-1b40cf4e49a6") : secret "node-exporter-tls" not found Jul 15 13:30:34.032754 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.032754 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032495 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-sys\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032754 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-textfile\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.032927 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.032909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.034565 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.034545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.034789 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.034768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.041140 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.041107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcb8\" (UniqueName: \"kubernetes.io/projected/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-kube-api-access-lpcb8\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.041545 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.041520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxjt\" (UniqueName: \"kubernetes.io/projected/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-api-access-kpxjt\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.436470 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.436434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:34.439123 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.439098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eda49ddc-ac2f-41ce-a01e-5c0efddf15f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-z6mn6\" (UID: \"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:34.537377 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.537338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:34.537558 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.537432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.537558 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:34.537521 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 13:30:34.537645 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:30:34.537611 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls podName:3299a86f-8ce9-442e-8c4d-1b40cf4e49a6 nodeName:}" failed. No retries permitted until 2026-07-15 13:30:35.53759326 +0000 UTC m=+77.055217437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls") pod "node-exporter-r2d5s" (UID: "3299a86f-8ce9-442e-8c4d-1b40cf4e49a6") : secret "node-exporter-tls" not found Jul 15 13:30:34.540195 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.540168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41cd44c2-5d29-46c9-ae23-b1efb285f8c7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-kg5vz\" (UID: \"41cd44c2-5d29-46c9-ae23-b1efb285f8c7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.585663 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.585629 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 13:30:34.588828 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.588806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.590954 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.590928 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 13:30:34.591186 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.591139 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 13:30:34.591371 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.591355 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 13:30:34.591587 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.591572 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 13:30:34.591751 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.591737 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 13:30:34.592149 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.592131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2bhcw\"" Jul 15 13:30:34.592332 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.592319 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 13:30:34.592520 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.592506 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 13:30:34.592672 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.592659 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 13:30:34.592847 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.592830 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 13:30:34.612036 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.612009 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 13:30:34.652982 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.652950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" Jul 15 13:30:34.678028 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.678005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.738880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55566a89-2540-4586-895e-758962b2fec6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.738947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrvg\" (UniqueName: \"kubernetes.io/projected/55566a89-2540-4586-895e-758962b2fec6-kube-api-access-knrvg\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.738984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55566a89-2540-4586-895e-758962b2fec6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55566a89-2540-4586-895e-758962b2fec6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55566a89-2540-4586-895e-758962b2fec6-config-out\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.739329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-web-config\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.742006 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.742006 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-config-volume\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.742006 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.739492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55566a89-2540-4586-895e-758962b2fec6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.807481 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.807450 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6"] Jul 15 13:30:34.810293 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:34.810268 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda49ddc_ac2f_41ce_a01e_5c0efddf15f8.slice/crio-fe5215baf3afdf1e8bdb56a7e0e73bb2433c482f9c08114f275644e0756886d9 WatchSource:0}: Error finding container fe5215baf3afdf1e8bdb56a7e0e73bb2433c482f9c08114f275644e0756886d9: Status 404 returned error can't find the container with id fe5215baf3afdf1e8bdb56a7e0e73bb2433c482f9c08114f275644e0756886d9 Jul 15 13:30:34.823778 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.823754 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-kg5vz"] Jul 15 13:30:34.826254 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:34.826214 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cd44c2_5d29_46c9_ae23_b1efb285f8c7.slice/crio-3324a1fb4d879b52b0e08270b390610a4d3cfa057562b373d091356ab22900f8 WatchSource:0}: Error finding container 3324a1fb4d879b52b0e08270b390610a4d3cfa057562b373d091356ab22900f8: Status 404 returned error can't find the container with id 3324a1fb4d879b52b0e08270b390610a4d3cfa057562b373d091356ab22900f8 Jul 15 13:30:34.840347 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55566a89-2540-4586-895e-758962b2fec6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840454 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55566a89-2540-4586-895e-758962b2fec6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840454 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrvg\" (UniqueName: \"kubernetes.io/projected/55566a89-2540-4586-895e-758962b2fec6-kube-api-access-knrvg\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840454 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840454 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840454 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840703 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55566a89-2540-4586-895e-758962b2fec6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840703 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55566a89-2540-4586-895e-758962b2fec6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840703 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55566a89-2540-4586-895e-758962b2fec6-config-out\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840703 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840703 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-web-config\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840703 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.840703 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-config-volume\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.841078 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.840870 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/55566a89-2540-4586-895e-758962b2fec6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.842160 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.842094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55566a89-2540-4586-895e-758962b2fec6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.842251 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.842221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/55566a89-2540-4586-895e-758962b2fec6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845090 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.843932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845090 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.844145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-web-config\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845090 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.844917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/55566a89-2540-4586-895e-758962b2fec6-config-out\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845090 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.844940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-config-volume\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845339 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.845304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/55566a89-2540-4586-895e-758962b2fec6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845339 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.845317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845476 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.845379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.845692 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.845665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.846294 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.846272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/55566a89-2540-4586-895e-758962b2fec6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.847812 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.847796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrvg\" (UniqueName: \"kubernetes.io/projected/55566a89-2540-4586-895e-758962b2fec6-kube-api-access-knrvg\") pod \"alertmanager-main-0\" (UID: \"55566a89-2540-4586-895e-758962b2fec6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:34.899503 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:34.899477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:30:35.059840 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.059721 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 13:30:35.062408 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:35.062368 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55566a89_2540_4586_895e_758962b2fec6.slice/crio-1e9a4065afc2dfbca5ddac38090b8e8c65002a16f35da98f57311bf06744aeae WatchSource:0}: Error finding container 1e9a4065afc2dfbca5ddac38090b8e8c65002a16f35da98f57311bf06744aeae: Status 404 returned error can't find the container with id 1e9a4065afc2dfbca5ddac38090b8e8c65002a16f35da98f57311bf06744aeae Jul 15 13:30:35.425204 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.425170 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" event={"ID":"41cd44c2-5d29-46c9-ae23-b1efb285f8c7","Type":"ContainerStarted","Data":"3324a1fb4d879b52b0e08270b390610a4d3cfa057562b373d091356ab22900f8"} Jul 15 13:30:35.427044 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.427012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" event={"ID":"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8","Type":"ContainerStarted","Data":"f498efe7cbb095dac3a921f2bc9680e61761f394cd7acb716aa8fa42afe208e5"} Jul 15 13:30:35.427212 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.427063 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" event={"ID":"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8","Type":"ContainerStarted","Data":"29e4cd688a402405012514029f6ef67ccc4d861f5cb0344613aa18a3426d7471"} Jul 15 13:30:35.427212 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.427083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" event={"ID":"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8","Type":"ContainerStarted","Data":"fe5215baf3afdf1e8bdb56a7e0e73bb2433c482f9c08114f275644e0756886d9"} Jul 15 13:30:35.428205 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.428182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerStarted","Data":"1e9a4065afc2dfbca5ddac38090b8e8c65002a16f35da98f57311bf06744aeae"} Jul 15 13:30:35.545264 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.545212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:35.548277 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.548248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3299a86f-8ce9-442e-8c4d-1b40cf4e49a6-node-exporter-tls\") pod \"node-exporter-r2d5s\" (UID: \"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6\") " pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:35.582555 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.582528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r2d5s" Jul 15 13:30:35.594414 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:35.594367 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3299a86f_8ce9_442e_8c4d_1b40cf4e49a6.slice/crio-87395c3656389678dba303471451fb601c9fe23037c1ed92c782718b34fe5359 WatchSource:0}: Error finding container 87395c3656389678dba303471451fb601c9fe23037c1ed92c782718b34fe5359: Status 404 returned error can't find the container with id 87395c3656389678dba303471451fb601c9fe23037c1ed92c782718b34fe5359 Jul 15 13:30:35.600249 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.600226 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6"] Jul 15 13:30:35.603740 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.603713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.606215 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.606193 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m92nq\"" Jul 15 13:30:35.606455 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.606432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 13:30:35.606564 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.606501 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 13:30:35.606626 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.606569 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dh0htrh0rreib\"" Jul 15 13:30:35.606704 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.606641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 13:30:35.607111 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.607093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 13:30:35.609108 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.608920 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 13:30:35.624312 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.624291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6"] Jul 15 13:30:35.747663 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.747663 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.747853 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.747853 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9whwk\" (UniqueName: \"kubernetes.io/projected/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-kube-api-access-9whwk\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.747853 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-metrics-client-ca\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.747853 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.748047 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-tls\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.748047 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.747962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-grpc-tls\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.848927 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.848893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.849389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.848950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-tls\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.849389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.848992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-grpc-tls\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.849389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.849026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.849389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.849073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.849389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.849099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.849389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.849160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whwk\" (UniqueName: \"kubernetes.io/projected/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-kube-api-access-9whwk\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.849389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.849189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-metrics-client-ca\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.850041 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.850014 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-metrics-client-ca\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.852426 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.852357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-grpc-tls\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.852591 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.852567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-tls\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.852752 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.852728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.853156 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.853133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.853265 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.853245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.854386 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.854364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.859541 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.859519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whwk\" (UniqueName: \"kubernetes.io/projected/5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8-kube-api-access-9whwk\") pod \"thanos-querier-7fc85f78c5-4wlq6\" (UID: \"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8\") " pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:35.915298 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:35.915264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:36.059811 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:36.059731 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:36.065245 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:36.065222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:30:36.433475 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:36.433420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2d5s" event={"ID":"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6","Type":"ContainerStarted","Data":"87395c3656389678dba303471451fb601c9fe23037c1ed92c782718b34fe5359"} Jul 15 13:30:36.708055 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:36.707688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6"] Jul 15 13:30:36.913916 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:36.913885 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea9df3b_bd08_4f7b_a082_bd13ff45f2f8.slice/crio-00e81f4d0688191b3fa1fbfc4857537436099675afed3a381a7f273f32461836 WatchSource:0}: Error finding container 00e81f4d0688191b3fa1fbfc4857537436099675afed3a381a7f273f32461836: Status 404 returned error can't find the container with id 00e81f4d0688191b3fa1fbfc4857537436099675afed3a381a7f273f32461836 Jul 15 13:30:37.440248 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.440214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" event={"ID":"41cd44c2-5d29-46c9-ae23-b1efb285f8c7","Type":"ContainerStarted","Data":"e58c311c8c883c4bb551fe8bd3b0fdae54983c570ac34ca9aaf1392840d760e6"} Jul 15 13:30:37.440430 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.440280 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" event={"ID":"41cd44c2-5d29-46c9-ae23-b1efb285f8c7","Type":"ContainerStarted","Data":"0c6917dc6f72021bc38c32642edb8ba531a71be911295add1482dc776731d045"} Jul 15 13:30:37.440430 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.440314 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" event={"ID":"41cd44c2-5d29-46c9-ae23-b1efb285f8c7","Type":"ContainerStarted","Data":"b6421e042629f47b1dbd26a2a54603516bf47f86269f3dea9517097508580b85"} Jul 15 13:30:37.442310 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.442285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" event={"ID":"eda49ddc-ac2f-41ce-a01e-5c0efddf15f8","Type":"ContainerStarted","Data":"e7c2b63a7e9da24a1afed6d8ff93f0961a47340263385d09d8881c9ce9ce84e3"} Jul 15 13:30:37.443674 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.443639 2576 generic.go:356] "Generic (PLEG): container finished" podID="55566a89-2540-4586-895e-758962b2fec6" containerID="e5bfd4106becaab9588c8e1f42b0b1b0e96e88504a68ef734be53de648012d36" exitCode=0 Jul 15 13:30:37.443778 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.443685 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerDied","Data":"e5bfd4106becaab9588c8e1f42b0b1b0e96e88504a68ef734be53de648012d36"} Jul 15 13:30:37.445361 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.445335 2576 generic.go:356] "Generic (PLEG): container finished" podID="3299a86f-8ce9-442e-8c4d-1b40cf4e49a6" containerID="1983397f7e4c06f72225c39d828b218e010096d1ac1320beae998b099d52e430" exitCode=0 Jul 15 13:30:37.445435 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.445392 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2d5s" event={"ID":"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6","Type":"ContainerDied","Data":"1983397f7e4c06f72225c39d828b218e010096d1ac1320beae998b099d52e430"} Jul 15 13:30:37.446577 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.446562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" event={"ID":"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8","Type":"ContainerStarted","Data":"00e81f4d0688191b3fa1fbfc4857537436099675afed3a381a7f273f32461836"} Jul 15 13:30:37.498005 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.497963 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-kg5vz" podStartSLOduration=2.789865368 podStartE2EDuration="4.497949749s" totalImagesPullingTime="1.708084381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:37.466768186 +0000 UTC m=+78.984392408" watchObservedRunningTime="2026-07-15 13:30:37.497949749 +0000 UTC m=+79.015573948" Jul 15 13:30:37.551967 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:37.551921 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-z6mn6" podStartSLOduration=2.962396097 podStartE2EDuration="4.551902719s" totalImagesPullingTime="1.589506622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:37.526217632 +0000 UTC m=+79.043841849" watchObservedRunningTime="2026-07-15 13:30:37.551902719 +0000 UTC m=+79.069526919" Jul 15 13:30:38.114187 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.114149 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5bd7dd65bc-swdqf"] Jul 15 13:30:38.116502 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.116482 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.119116 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.119092 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-98npk71pn1kqf\"" Jul 15 13:30:38.119116 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.119110 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pc6d7\"" Jul 15 13:30:38.119286 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.119096 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 13:30:38.119286 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.119219 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 13:30:38.119617 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.119599 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 13:30:38.119737 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.119658 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 13:30:38.129180 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.129159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bd7dd65bc-swdqf"] Jul 15 13:30:38.264146 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.264116 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb"] Jul 15 13:30:38.266780 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.266435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.268660 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.268624 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 13:30:38.268849 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.268829 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v9cj6\"" Jul 15 13:30:38.271979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.271737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-secret-metrics-server-client-certs\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.271979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.271776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-client-ca-bundle\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.271979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.271856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d64398d2-26df-4757-a2ac-65cb057b74e4-metrics-server-audit-profiles\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.271979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.271886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzdg\" (UniqueName: \"kubernetes.io/projected/d64398d2-26df-4757-a2ac-65cb057b74e4-kube-api-access-vgzdg\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.271979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.271915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d64398d2-26df-4757-a2ac-65cb057b74e4-audit-log\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.271979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.271953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-secret-metrics-server-tls\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.272327 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.272033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64398d2-26df-4757-a2ac-65cb057b74e4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.282672 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.282649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb"] Jul 15 13:30:38.372936 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.372854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/182235c3-4fba-4aac-b1a8-33834d39d0c9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-zjsgb\" (UID: \"182235c3-4fba-4aac-b1a8-33834d39d0c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.372936 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.372913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-secret-metrics-server-client-certs\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.372939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-client-ca-bundle\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.372972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5j8t\" (UniqueName: \"kubernetes.io/projected/182235c3-4fba-4aac-b1a8-33834d39d0c9-kube-api-access-n5j8t\") pod \"monitoring-plugin-5f68df7f58-zjsgb\" (UID: \"182235c3-4fba-4aac-b1a8-33834d39d0c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.373147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.373011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d64398d2-26df-4757-a2ac-65cb057b74e4-metrics-server-audit-profiles\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.373035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzdg\" (UniqueName: \"kubernetes.io/projected/d64398d2-26df-4757-a2ac-65cb057b74e4-kube-api-access-vgzdg\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.373060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d64398d2-26df-4757-a2ac-65cb057b74e4-audit-log\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.373091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-secret-metrics-server-tls\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.373135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64398d2-26df-4757-a2ac-65cb057b74e4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373694 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.373666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d64398d2-26df-4757-a2ac-65cb057b74e4-audit-log\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.373861 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.373844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d64398d2-26df-4757-a2ac-65cb057b74e4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.374051 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.374034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d64398d2-26df-4757-a2ac-65cb057b74e4-metrics-server-audit-profiles\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.375710 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.375686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-secret-metrics-server-tls\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.375817 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.375800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-client-ca-bundle\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.375911 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.375895 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d64398d2-26df-4757-a2ac-65cb057b74e4-secret-metrics-server-client-certs\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.381618 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.381598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzdg\" (UniqueName: \"kubernetes.io/projected/d64398d2-26df-4757-a2ac-65cb057b74e4-kube-api-access-vgzdg\") pod \"metrics-server-5bd7dd65bc-swdqf\" (UID: \"d64398d2-26df-4757-a2ac-65cb057b74e4\") " pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.427596 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.427569 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:38.457333 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.457226 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2d5s" event={"ID":"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6","Type":"ContainerStarted","Data":"3936fd643782253e79e3c237fb2cd579b342a4ed0397e79957dc6a4749819af7"} Jul 15 13:30:38.457333 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.457289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2d5s" event={"ID":"3299a86f-8ce9-442e-8c4d-1b40cf4e49a6","Type":"ContainerStarted","Data":"23f952c1a16481be96e2843b301f2400af7267d4464e5169889745b27dc44a09"} Jul 15 13:30:38.474030 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.473995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5j8t\" (UniqueName: \"kubernetes.io/projected/182235c3-4fba-4aac-b1a8-33834d39d0c9-kube-api-access-n5j8t\") pod \"monitoring-plugin-5f68df7f58-zjsgb\" (UID: \"182235c3-4fba-4aac-b1a8-33834d39d0c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.474152 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.474131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/182235c3-4fba-4aac-b1a8-33834d39d0c9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-zjsgb\" (UID: \"182235c3-4fba-4aac-b1a8-33834d39d0c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.478038 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.478009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/182235c3-4fba-4aac-b1a8-33834d39d0c9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-zjsgb\" (UID: \"182235c3-4fba-4aac-b1a8-33834d39d0c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.482585 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.482505 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r2d5s" podStartSLOduration=4.072172232 podStartE2EDuration="5.48248596s" totalImagesPullingTime="1.345987899s" totalInitContainerRuntime="64.325829ms" isStatefulPod=true podCreationTimestamp="2026-07-15 13:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:38.481118721 +0000 UTC m=+79.998742919" watchObservedRunningTime="2026-07-15 13:30:38.48248596 +0000 UTC m=+80.000110161" Jul 15 13:30:38.489289 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.489238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5j8t\" (UniqueName: \"kubernetes.io/projected/182235c3-4fba-4aac-b1a8-33834d39d0c9-kube-api-access-n5j8t\") pod \"monitoring-plugin-5f68df7f58-zjsgb\" (UID: \"182235c3-4fba-4aac-b1a8-33834d39d0c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.583691 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.583663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:38.607658 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.607620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bd7dd65bc-swdqf"] Jul 15 13:30:38.839373 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:38.839295 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64398d2_26df_4757_a2ac_65cb057b74e4.slice/crio-4a02fb9b222a6ab858db7eaebe9ced4c3e3bd3df1e65e117d416dfca1c48cc54 WatchSource:0}: Error finding container 4a02fb9b222a6ab858db7eaebe9ced4c3e3bd3df1e65e117d416dfca1c48cc54: Status 404 returned error can't find the container with id 4a02fb9b222a6ab858db7eaebe9ced4c3e3bd3df1e65e117d416dfca1c48cc54 Jul 15 13:30:38.942924 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.942888 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b9c794874-wmm96"] Jul 15 13:30:38.945447 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.945422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:38.950185 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.950163 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 13:30:38.950461 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.950445 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 13:30:38.951447 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.951016 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 13:30:38.951447 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.951117 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 13:30:38.951447 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.951266 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ns9h9\"" Jul 15 13:30:38.951447 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.951317 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 13:30:38.956945 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.956925 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 13:30:38.963789 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:38.963766 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b9c794874-wmm96"] Jul 15 13:30:39.080113 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8np\" (UniqueName: \"kubernetes.io/projected/6fc6e973-642a-4001-9011-38705d5dbce0-kube-api-access-fr8np\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.080280 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-secret-telemeter-client\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.080334 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.080382 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-serving-certs-ca-bundle\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.080467 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-metrics-client-ca\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.080522 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.080565 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-telemeter-client-tls\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.080608 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.080573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-federate-client-tls\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182068 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-metrics-client-ca\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-telemeter-client-tls\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-federate-client-tls\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8np\" (UniqueName: \"kubernetes.io/projected/6fc6e973-642a-4001-9011-38705d5dbce0-kube-api-access-fr8np\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-secret-telemeter-client\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-serving-certs-ca-bundle\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182917 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-metrics-client-ca\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.182966 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.182940 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-serving-certs-ca-bundle\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.183782 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.183751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fc6e973-642a-4001-9011-38705d5dbce0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.185148 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.185117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.185551 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.185530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-telemeter-client-tls\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.185632 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.185598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-secret-telemeter-client\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.185632 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.185624 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6fc6e973-642a-4001-9011-38705d5dbce0-federate-client-tls\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.192709 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.192690 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8np\" (UniqueName: \"kubernetes.io/projected/6fc6e973-642a-4001-9011-38705d5dbce0-kube-api-access-fr8np\") pod \"telemeter-client-5b9c794874-wmm96\" (UID: \"6fc6e973-642a-4001-9011-38705d5dbce0\") " pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.256071 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.256028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" Jul 15 13:30:39.464522 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.464454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" event={"ID":"d64398d2-26df-4757-a2ac-65cb057b74e4","Type":"ContainerStarted","Data":"4a02fb9b222a6ab858db7eaebe9ced4c3e3bd3df1e65e117d416dfca1c48cc54"} Jul 15 13:30:39.575647 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.575619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb"] Jul 15 13:30:39.580696 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:39.580477 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182235c3_4fba_4aac_b1a8_33834d39d0c9.slice/crio-237cfe5f0984a1ba0639f1c6399417d9e97623f8c6cf9c46d781cefed09b9036 WatchSource:0}: Error finding container 237cfe5f0984a1ba0639f1c6399417d9e97623f8c6cf9c46d781cefed09b9036: Status 404 returned error can't find the container with id 237cfe5f0984a1ba0639f1c6399417d9e97623f8c6cf9c46d781cefed09b9036 Jul 15 13:30:39.608869 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:39.608741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b9c794874-wmm96"] Jul 15 13:30:39.615632 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:39.615601 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc6e973_642a_4001_9011_38705d5dbce0.slice/crio-760da27b9da2fae4e82047a34604be951a7454a74c29748f7e3743ad5d5af9a1 WatchSource:0}: Error finding container 760da27b9da2fae4e82047a34604be951a7454a74c29748f7e3743ad5d5af9a1: Status 404 returned error can't find the container with id 760da27b9da2fae4e82047a34604be951a7454a74c29748f7e3743ad5d5af9a1 Jul 15 13:30:40.046723 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.046123 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 13:30:40.050261 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.050239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.057610 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.057332 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 13:30:40.057610 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.057500 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 13:30:40.058224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.057882 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 13:30:40.058224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.057937 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 13:30:40.058224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.058119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 13:30:40.058595 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.058573 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 13:30:40.058679 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.058621 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 13:30:40.058730 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.058579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 13:30:40.062420 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.062383 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 13:30:40.062622 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.062607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 13:30:40.063356 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.062905 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bs301hi3dchne\"" Jul 15 13:30:40.063356 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.063195 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hscr9\"" Jul 15 13:30:40.063356 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.063288 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 13:30:40.063603 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.063585 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 13:30:40.067167 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.067147 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 13:30:40.081249 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.081217 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 13:30:40.086602 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.084412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 13:30:40.191539 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3eb5954e-f956-444a-9960-9db974faa315-config-out\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3eb5954e-f956-444a-9960-9db974faa315-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-config\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-web-config\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvnx\" (UniqueName: \"kubernetes.io/projected/3eb5954e-f956-444a-9960-9db974faa315-kube-api-access-jvvnx\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.191980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192008 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.192016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.192043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.192071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.192102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.192255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.192664 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.192346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293629 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293629 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3eb5954e-f956-444a-9960-9db974faa315-config-out\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3eb5954e-f956-444a-9960-9db974faa315-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-config\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-web-config\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvnx\" (UniqueName: \"kubernetes.io/projected/3eb5954e-f956-444a-9960-9db974faa315-kube-api-access-jvvnx\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.293970 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.293976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.294519 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.294519 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.294519 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.294519 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.294746 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294719 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.298387 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.298387 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.298387 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.298387 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.294954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.298387 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.295341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.298387 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.295444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.298387 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.297445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.299793 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.299762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-config\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.300693 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.300639 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3eb5954e-f956-444a-9960-9db974faa315-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.301345 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.300961 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.301345 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.301297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.301537 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.301351 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.302080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.302034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.302278 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.302253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.302891 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.302865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3eb5954e-f956-444a-9960-9db974faa315-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.303497 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.303473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3eb5954e-f956-444a-9960-9db974faa315-config-out\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.303679 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.303634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.304183 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.303807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.304183 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.304143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.304564 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.304542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.305338 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.305292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3eb5954e-f956-444a-9960-9db974faa315-web-config\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.309911 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.309866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvnx\" (UniqueName: \"kubernetes.io/projected/3eb5954e-f956-444a-9960-9db974faa315-kube-api-access-jvvnx\") pod \"prometheus-k8s-0\" (UID: \"3eb5954e-f956-444a-9960-9db974faa315\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.364554 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.364480 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:40.475180 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.475133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerStarted","Data":"1bd84e15a2253c7282f550aad9e8fb82570c50de4c1d3f72e40cea7d68194cce"} Jul 15 13:30:40.475360 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.475219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerStarted","Data":"39e3157cbbc4bb116beaa6b8f5960ea50ca1728d523a9c6d43862f7e765f18ca"} Jul 15 13:30:40.475360 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.475261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerStarted","Data":"e7cedad42830f6aacfc18a522e1ec3e5711a219a0aa78a303af6d7a686b65f5a"} Jul 15 13:30:40.475360 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.475301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerStarted","Data":"4b4f3421cdc845b8ea2ebcebf5d2fe18c6c045fd7b133df6f991136ff88fc3eb"} Jul 15 13:30:40.475360 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.475339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerStarted","Data":"1da2c1c81b1b38cab1bae805c95df9fb580c103a16ba8392e8da02eddf411aa1"} Jul 15 13:30:40.476630 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.476580 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" event={"ID":"182235c3-4fba-4aac-b1a8-33834d39d0c9","Type":"ContainerStarted","Data":"237cfe5f0984a1ba0639f1c6399417d9e97623f8c6cf9c46d781cefed09b9036"} Jul 15 13:30:40.479531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.479492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" event={"ID":"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8","Type":"ContainerStarted","Data":"ee35db35bc43784cdf132f555e29174b5deec5c7d7152458b8b4115dcbbbe4b5"} Jul 15 13:30:40.479667 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.479568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" event={"ID":"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8","Type":"ContainerStarted","Data":"46db66dad381980cc92d7884d0f2809ec2f613523dbe451b244fb26fe46d0dee"} Jul 15 13:30:40.479667 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.479610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" event={"ID":"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8","Type":"ContainerStarted","Data":"09fc23c0c899ee6807c81bb74e9e3266e835dd23e4ba2b1fc5dd362cf0af61d1"} Jul 15 13:30:40.480988 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:40.480958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" event={"ID":"6fc6e973-642a-4001-9011-38705d5dbce0","Type":"ContainerStarted","Data":"760da27b9da2fae4e82047a34604be951a7454a74c29748f7e3743ad5d5af9a1"} Jul 15 13:30:41.366877 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.366768 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 13:30:41.369502 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:30:41.369461 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb5954e_f956_444a_9960_9db974faa315.slice/crio-59b9f1c748fe46ca5810d3a14406ec2331698e526f9a6a2da4d11dfdc180c965 WatchSource:0}: Error finding container 59b9f1c748fe46ca5810d3a14406ec2331698e526f9a6a2da4d11dfdc180c965: Status 404 returned error can't find the container with id 59b9f1c748fe46ca5810d3a14406ec2331698e526f9a6a2da4d11dfdc180c965 Jul 15 13:30:41.488490 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.488452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"55566a89-2540-4586-895e-758962b2fec6","Type":"ContainerStarted","Data":"48dfc652163c0b9915328143ffba75015edc43edd3018af5f90d57e8fbcd2975"} Jul 15 13:30:41.489786 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.489766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" event={"ID":"182235c3-4fba-4aac-b1a8-33834d39d0c9","Type":"ContainerStarted","Data":"8f6bd58ef39b512bdf31129f7e41ef5b909cbab3e465cefefd117b850c7eb6b8"} Jul 15 13:30:41.490195 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.490178 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:41.495691 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.495674 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" Jul 15 13:30:41.498547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.498528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" event={"ID":"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8","Type":"ContainerStarted","Data":"35ab39a43cb19bf51a3ed9191a5a791c606d93daf701ee7d7b7479e18666955b"} Jul 15 13:30:41.498686 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.498573 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" event={"ID":"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8","Type":"ContainerStarted","Data":"408d21896ee9dbebe9eeaaf48a9b82c5641c28fa46a4fbd7c8a55a51ab74910c"} Jul 15 13:30:41.498686 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.498610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" event={"ID":"5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8","Type":"ContainerStarted","Data":"6ee60f0f411878257c676d15dfecd74230d2e41a760644a6eb03b867c910badd"} Jul 15 13:30:41.498686 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.498650 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:41.500094 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.500073 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"631a65bd788c9a2c36ec286150dd9d0817f359b49b7696068123955ebf49cf32"} Jul 15 13:30:41.500181 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.500134 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"59b9f1c748fe46ca5810d3a14406ec2331698e526f9a6a2da4d11dfdc180c965"} Jul 15 13:30:41.501840 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.501536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" event={"ID":"d64398d2-26df-4757-a2ac-65cb057b74e4","Type":"ContainerStarted","Data":"bc656b57b032ab27695fd9f4e87b43eca8f8a7f4ce3a6dae15a821511ed0a6b7"} Jul 15 13:30:41.531648 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.531602 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.709207845 podStartE2EDuration="7.531586072s" totalImagesPullingTime="4.797782679s" totalInitContainerRuntime="24.595548ms" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:41.529801205 +0000 UTC m=+83.047425402" watchObservedRunningTime="2026-07-15 13:30:41.531586072 +0000 UTC m=+83.049210271" Jul 15 13:30:41.624162 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.624099 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" podStartSLOduration=2.623908181 podStartE2EDuration="6.624078324s" totalImagesPullingTime="4.000170143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:41.622480514 +0000 UTC m=+83.140104714" watchObservedRunningTime="2026-07-15 13:30:41.624078324 +0000 UTC m=+83.141702522" Jul 15 13:30:41.678907 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.678808 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" podStartSLOduration=1.352689285 podStartE2EDuration="3.678783411s" totalImagesPullingTime="2.326094126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:41.651026072 +0000 UTC m=+83.168650295" watchObservedRunningTime="2026-07-15 13:30:41.678783411 +0000 UTC m=+83.196407611" Jul 15 13:30:41.679906 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:41.679867 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-zjsgb" podStartSLOduration=2.093692421 podStartE2EDuration="3.67985548s" totalImagesPullingTime="1.586163059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:41.677845566 +0000 UTC m=+83.195469787" watchObservedRunningTime="2026-07-15 13:30:41.67985548 +0000 UTC m=+83.197479682" Jul 15 13:30:42.508075 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:42.508033 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" event={"ID":"6fc6e973-642a-4001-9011-38705d5dbce0","Type":"ContainerStarted","Data":"0650e6d4e85d486fd3a3c227dc79fb74f3d5ff490d442873abff8d46da547b8c"} Jul 15 13:30:42.508515 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:42.508133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" event={"ID":"6fc6e973-642a-4001-9011-38705d5dbce0","Type":"ContainerStarted","Data":"6481b2dc5c71e12ac76fd2085fc93a0646194472a34057284fb1ac59a1124060"} Jul 15 13:30:42.508515 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:42.508162 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" event={"ID":"6fc6e973-642a-4001-9011-38705d5dbce0","Type":"ContainerStarted","Data":"c4448153cd0dca09b17e0883476e40130402c8eaf1b8f56efb5e95b7bfc21939"} Jul 15 13:30:42.509554 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:42.509535 2576 generic.go:356] "Generic (PLEG): container finished" podID="3eb5954e-f956-444a-9960-9db974faa315" containerID="631a65bd788c9a2c36ec286150dd9d0817f359b49b7696068123955ebf49cf32" exitCode=0 Jul 15 13:30:42.509657 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:42.509634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerDied","Data":"631a65bd788c9a2c36ec286150dd9d0817f359b49b7696068123955ebf49cf32"} Jul 15 13:30:42.538505 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:42.538460 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b9c794874-wmm96" podStartSLOduration=2.1563903189999998 podStartE2EDuration="4.538440649s" totalImagesPullingTime="2.38205033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:42.536935663 +0000 UTC m=+84.054559872" watchObservedRunningTime="2026-07-15 13:30:42.538440649 +0000 UTC m=+84.056064847" Jul 15 13:30:46.533016 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:46.532981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"5a2413d446be16ecbec01b7aee9e11cc508cc8780585a1281a86a24955887866"} Jul 15 13:30:46.533386 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:46.533048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"cf9e4bd1d2ed8e14bc480f3c685c315d261fdabed81eb1335d84a6df40ccc32c"} Jul 15 13:30:46.533386 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:46.533079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"d06a57cd8554632920987a7395e02c3597d9d030ab3e204ed58c1a753eb4a0e4"} Jul 15 13:30:46.533386 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:46.533111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"052e4be7af24dd403fb4aebc9f4719a4fb582e51a05fd37380285de31c856001"} Jul 15 13:30:46.533386 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:46.533140 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"cc4794ae1165eb4c21d21bbb987061ba24fac40d08553ed40849cf2b3b23e63f"} Jul 15 13:30:46.533386 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:46.533164 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3eb5954e-f956-444a-9960-9db974faa315","Type":"ContainerStarted","Data":"0d1002f1e03895921eeb0b1e94b44b7c8ee446d546ad979ebe9bc3f140f36dbe"} Jul 15 13:30:47.515754 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:47.515725 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fc85f78c5-4wlq6" Jul 15 13:30:47.560978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:47.560920 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.288462136 podStartE2EDuration="7.560903809s" totalImagesPullingTime="3.249325143s" totalInitContainerRuntime="23.11653ms" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:46.57544536 +0000 UTC m=+88.093069559" watchObservedRunningTime="2026-07-15 13:30:47.560903809 +0000 UTC m=+89.078528008" Jul 15 13:30:50.365547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:50.365505 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:30:54.148974 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:54.148943 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86799ff95b-mnbb5"] Jul 15 13:30:58.428045 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:58.428005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:30:58.428491 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:30:58.428070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:31:02.416625 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:02.416594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rbsfv" Jul 15 13:31:18.433056 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:18.433022 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:31:18.436845 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:18.436827 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bd7dd65bc-swdqf" Jul 15 13:31:19.169751 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.169696 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86799ff95b-mnbb5" podUID="70479f4d-58a4-45b4-8870-0e84b7d4eec2" containerName="console" containerID="cri-o://7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b" gracePeriod=15 Jul 15 13:31:19.199185 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:31:19.199153 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70479f4d_58a4_45b4_8870_0e84b7d4eec2.slice/crio-7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b.scope\": RecentStats: unable to find data in memory cache]" Jul 15 13:31:19.435722 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.435695 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:31:19.544699 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.544659 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-oauth-serving-cert\") pod \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " Jul 15 13:31:19.544879 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.544721 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-trusted-ca-bundle\") pod \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " Jul 15 13:31:19.544879 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.544767 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-serving-cert\") pod \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " Jul 15 13:31:19.544879 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.544811 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-service-ca\") pod \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " Jul 15 13:31:19.544879 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.544842 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdhv6\" (UniqueName: \"kubernetes.io/projected/70479f4d-58a4-45b4-8870-0e84b7d4eec2-kube-api-access-fdhv6\") pod \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " Jul 15 13:31:19.544879 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.544868 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-config\") pod \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " Jul 15 13:31:19.545136 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.544895 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-oauth-config\") pod \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\" (UID: \"70479f4d-58a4-45b4-8870-0e84b7d4eec2\") " Jul 15 13:31:19.545136 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.545010 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "70479f4d-58a4-45b4-8870-0e84b7d4eec2" (UID: "70479f4d-58a4-45b4-8870-0e84b7d4eec2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:31:19.545240 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.545170 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-oauth-serving-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:31:19.545240 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.545212 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "70479f4d-58a4-45b4-8870-0e84b7d4eec2" (UID: "70479f4d-58a4-45b4-8870-0e84b7d4eec2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:31:19.545240 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.545220 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-service-ca" (OuterVolumeSpecName: "service-ca") pod "70479f4d-58a4-45b4-8870-0e84b7d4eec2" (UID: "70479f4d-58a4-45b4-8870-0e84b7d4eec2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:31:19.545347 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.545284 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-config" (OuterVolumeSpecName: "console-config") pod "70479f4d-58a4-45b4-8870-0e84b7d4eec2" (UID: "70479f4d-58a4-45b4-8870-0e84b7d4eec2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:31:19.547064 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.547036 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "70479f4d-58a4-45b4-8870-0e84b7d4eec2" (UID: "70479f4d-58a4-45b4-8870-0e84b7d4eec2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:31:19.547168 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.547125 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "70479f4d-58a4-45b4-8870-0e84b7d4eec2" (UID: "70479f4d-58a4-45b4-8870-0e84b7d4eec2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:31:19.547205 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.547160 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70479f4d-58a4-45b4-8870-0e84b7d4eec2-kube-api-access-fdhv6" (OuterVolumeSpecName: "kube-api-access-fdhv6") pod "70479f4d-58a4-45b4-8870-0e84b7d4eec2" (UID: "70479f4d-58a4-45b4-8870-0e84b7d4eec2"). InnerVolumeSpecName "kube-api-access-fdhv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:31:19.645649 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.645612 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-serving-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:31:19.645649 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.645643 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:31:19.645649 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.645653 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhv6\" (UniqueName: \"kubernetes.io/projected/70479f4d-58a4-45b4-8870-0e84b7d4eec2-kube-api-access-fdhv6\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:31:19.645868 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.645662 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-config\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:31:19.645868 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.645670 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70479f4d-58a4-45b4-8870-0e84b7d4eec2-console-oauth-config\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:31:19.645868 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.645678 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70479f4d-58a4-45b4-8870-0e84b7d4eec2-trusted-ca-bundle\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:31:19.674495 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.674422 2576 generic.go:356] "Generic (PLEG): container finished" podID="70479f4d-58a4-45b4-8870-0e84b7d4eec2" containerID="7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b" exitCode=0 Jul 15 13:31:19.674495 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.674478 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86799ff95b-mnbb5" Jul 15 13:31:19.674639 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.674493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86799ff95b-mnbb5" event={"ID":"70479f4d-58a4-45b4-8870-0e84b7d4eec2","Type":"ContainerDied","Data":"7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b"} Jul 15 13:31:19.674639 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.674542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86799ff95b-mnbb5" event={"ID":"70479f4d-58a4-45b4-8870-0e84b7d4eec2","Type":"ContainerDied","Data":"32e4f17e503d89caaabda5fadef036513891df66e454b6bd852c6f136ceb7d56"} Jul 15 13:31:19.674639 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.674569 2576 scope.go:117] "RemoveContainer" containerID="7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b" Jul 15 13:31:19.683257 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.683236 2576 scope.go:117] "RemoveContainer" containerID="7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b" Jul 15 13:31:19.683552 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:31:19.683532 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b\": container with ID starting with 7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b not found: ID does not exist" containerID="7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b" Jul 15 13:31:19.683618 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.683560 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b"} err="failed to get container status \"7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b\": rpc error: code = NotFound desc = could not find container \"7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b\": container with ID starting with 7f8283900fffdcae41176115fc9b950558725e0c74244289e0e14e021de8142b not found: ID does not exist" Jul 15 13:31:19.701572 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.701545 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86799ff95b-mnbb5"] Jul 15 13:31:19.706614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:19.706592 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86799ff95b-mnbb5"] Jul 15 13:31:21.095652 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:21.095611 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70479f4d-58a4-45b4-8870-0e84b7d4eec2" path="/var/lib/kubelet/pods/70479f4d-58a4-45b4-8870-0e84b7d4eec2/volumes" Jul 15 13:31:40.364929 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:40.364881 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:31:40.384877 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:40.384847 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:31:40.776166 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:31:40.776087 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:33:52.286632 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.286554 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p95sx"] Jul 15 13:33:52.287070 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.286944 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70479f4d-58a4-45b4-8870-0e84b7d4eec2" containerName="console" Jul 15 13:33:52.287070 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.286961 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70479f4d-58a4-45b4-8870-0e84b7d4eec2" containerName="console" Jul 15 13:33:52.287070 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.287028 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="70479f4d-58a4-45b4-8870-0e84b7d4eec2" containerName="console" Jul 15 13:33:52.288937 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.288921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.292304 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.292283 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 13:33:52.292467 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.292369 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 13:33:52.292736 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.292715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4kc6z\"" Jul 15 13:33:52.299809 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.299789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p95sx"] Jul 15 13:33:52.440659 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.440631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e69bf459-adf4-4678-a570-dc85d154e25f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p95sx\" (UID: \"e69bf459-adf4-4678-a570-dc85d154e25f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.440825 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.440674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbfx\" (UniqueName: \"kubernetes.io/projected/e69bf459-adf4-4678-a570-dc85d154e25f-kube-api-access-ppbfx\") pod \"cert-manager-cainjector-559cdd8c87-p95sx\" (UID: \"e69bf459-adf4-4678-a570-dc85d154e25f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.541745 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.541661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e69bf459-adf4-4678-a570-dc85d154e25f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p95sx\" (UID: \"e69bf459-adf4-4678-a570-dc85d154e25f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.541745 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.541704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbfx\" (UniqueName: \"kubernetes.io/projected/e69bf459-adf4-4678-a570-dc85d154e25f-kube-api-access-ppbfx\") pod \"cert-manager-cainjector-559cdd8c87-p95sx\" (UID: \"e69bf459-adf4-4678-a570-dc85d154e25f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.549475 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.549444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e69bf459-adf4-4678-a570-dc85d154e25f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p95sx\" (UID: \"e69bf459-adf4-4678-a570-dc85d154e25f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.549609 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.549569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbfx\" (UniqueName: \"kubernetes.io/projected/e69bf459-adf4-4678-a570-dc85d154e25f-kube-api-access-ppbfx\") pod \"cert-manager-cainjector-559cdd8c87-p95sx\" (UID: \"e69bf459-adf4-4678-a570-dc85d154e25f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.605979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.605945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" Jul 15 13:33:52.736732 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:52.736708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p95sx"] Jul 15 13:33:52.739373 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:33:52.739348 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69bf459_adf4_4678_a570_dc85d154e25f.slice/crio-d00c911cfbd2b3f25a24ee24211216d4d325823f06c2fe8a2b44f20e56bdac7e WatchSource:0}: Error finding container d00c911cfbd2b3f25a24ee24211216d4d325823f06c2fe8a2b44f20e56bdac7e: Status 404 returned error can't find the container with id d00c911cfbd2b3f25a24ee24211216d4d325823f06c2fe8a2b44f20e56bdac7e Jul 15 13:33:53.274613 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:53.274577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" event={"ID":"e69bf459-adf4-4678-a570-dc85d154e25f","Type":"ContainerStarted","Data":"d00c911cfbd2b3f25a24ee24211216d4d325823f06c2fe8a2b44f20e56bdac7e"} Jul 15 13:33:56.290285 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:56.290241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" event={"ID":"e69bf459-adf4-4678-a570-dc85d154e25f","Type":"ContainerStarted","Data":"6c88786130c9e3761e676c715a26d949ef7b0a0221e3ad7b2f97157aa2a1ed18"} Jul 15 13:33:56.307657 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:33:56.307599 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p95sx" podStartSLOduration=1.424592027 podStartE2EDuration="4.30757893s" totalImagesPullingTime="2.882986903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:33:56.307237525 +0000 UTC m=+277.824861727" watchObservedRunningTime="2026-07-15 13:33:56.30757893 +0000 UTC m=+277.825203134" Jul 15 13:34:00.772252 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:00.772221 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ddqqk"] Jul 15 13:34:00.775124 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:00.775102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:00.777237 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:00.777215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5lrcw\"" Jul 15 13:34:00.788069 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:00.788051 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ddqqk"] Jul 15 13:34:00.911639 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:00.911604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b387ab68-7463-4cb0-8030-d1e1bedf057d-bound-sa-token\") pod \"cert-manager-55b4d954d5-ddqqk\" (UID: \"b387ab68-7463-4cb0-8030-d1e1bedf057d\") " pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:00.911796 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:00.911673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxt9\" (UniqueName: \"kubernetes.io/projected/b387ab68-7463-4cb0-8030-d1e1bedf057d-kube-api-access-lxxt9\") pod \"cert-manager-55b4d954d5-ddqqk\" (UID: \"b387ab68-7463-4cb0-8030-d1e1bedf057d\") " pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:01.012332 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.012299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxt9\" (UniqueName: \"kubernetes.io/projected/b387ab68-7463-4cb0-8030-d1e1bedf057d-kube-api-access-lxxt9\") pod \"cert-manager-55b4d954d5-ddqqk\" (UID: \"b387ab68-7463-4cb0-8030-d1e1bedf057d\") " pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:01.012513 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.012368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b387ab68-7463-4cb0-8030-d1e1bedf057d-bound-sa-token\") pod \"cert-manager-55b4d954d5-ddqqk\" (UID: \"b387ab68-7463-4cb0-8030-d1e1bedf057d\") " pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:01.020698 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.020671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b387ab68-7463-4cb0-8030-d1e1bedf057d-bound-sa-token\") pod \"cert-manager-55b4d954d5-ddqqk\" (UID: \"b387ab68-7463-4cb0-8030-d1e1bedf057d\") " pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:01.020804 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.020725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxt9\" (UniqueName: \"kubernetes.io/projected/b387ab68-7463-4cb0-8030-d1e1bedf057d-kube-api-access-lxxt9\") pod \"cert-manager-55b4d954d5-ddqqk\" (UID: \"b387ab68-7463-4cb0-8030-d1e1bedf057d\") " pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:01.083984 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.083925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ddqqk" Jul 15 13:34:01.207598 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.207467 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ddqqk"] Jul 15 13:34:01.209887 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:34:01.209863 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb387ab68_7463_4cb0_8030_d1e1bedf057d.slice/crio-a0434ac97845b49a6f64246b0e84eebde6116a59236699cb75064e76aada06cf WatchSource:0}: Error finding container a0434ac97845b49a6f64246b0e84eebde6116a59236699cb75064e76aada06cf: Status 404 returned error can't find the container with id a0434ac97845b49a6f64246b0e84eebde6116a59236699cb75064e76aada06cf Jul 15 13:34:01.312487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.312450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ddqqk" event={"ID":"b387ab68-7463-4cb0-8030-d1e1bedf057d","Type":"ContainerStarted","Data":"f57a895ddcbad8216fa36587ccc21ff68f795a3ef671c5fff8fc6f223886e6a3"} Jul 15 13:34:01.312668 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.312499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ddqqk" event={"ID":"b387ab68-7463-4cb0-8030-d1e1bedf057d","Type":"ContainerStarted","Data":"a0434ac97845b49a6f64246b0e84eebde6116a59236699cb75064e76aada06cf"} Jul 15 13:34:01.335438 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:01.335313 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ddqqk" podStartSLOduration=1.335294265 podStartE2EDuration="1.335294265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:34:01.334042051 +0000 UTC m=+282.851666264" watchObservedRunningTime="2026-07-15 13:34:01.335294265 +0000 UTC m=+282.852918467" Jul 15 13:34:18.860487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:18.860457 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:34:18.861275 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:18.861250 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:34:22.781896 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.781840 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w"] Jul 15 13:34:22.783988 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.783970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.795555 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.795534 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 13:34:22.795670 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.795649 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 13:34:22.795763 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.795749 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 13:34:22.795888 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.795873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cp25j\"" Jul 15 13:34:22.801547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.801534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 13:34:22.821388 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.821362 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w"] Jul 15 13:34:22.896239 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.896212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-manifests\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.896421 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.896245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.896421 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.896266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.896421 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.896375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2cx\" (UniqueName: \"kubernetes.io/projected/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-kube-api-access-kt2cx\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.896421 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.896418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-tmp\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.997509 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.997471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2cx\" (UniqueName: \"kubernetes.io/projected/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-kube-api-access-kt2cx\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.997787 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.997516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-tmp\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.997787 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.997588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-manifests\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.997787 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.997623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.997787 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.997649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.998019 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.997952 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-manifests\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.999774 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.999753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-tmp\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:22.999981 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:22.999963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:23.000085 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:23.000066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:23.004796 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:23.004778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2cx\" (UniqueName: \"kubernetes.io/projected/7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1-kube-api-access-kt2cx\") pod \"opendatahub-operator-controller-manager-84978f4d9d-5px9w\" (UID: \"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1\") " pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:23.093584 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:23.093505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:23.233589 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:23.233553 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w"] Jul 15 13:34:23.236504 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:34:23.236473 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2c3e2e_6eb1_472d_8a2e_cf416b2ec7e1.slice/crio-a712d2fdaa7ff39cf1988d065668d30195e43f9806e6684f0c2cf87e6405d81d WatchSource:0}: Error finding container a712d2fdaa7ff39cf1988d065668d30195e43f9806e6684f0c2cf87e6405d81d: Status 404 returned error can't find the container with id a712d2fdaa7ff39cf1988d065668d30195e43f9806e6684f0c2cf87e6405d81d Jul 15 13:34:23.238531 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:23.238510 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:34:23.397750 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:23.397716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" event={"ID":"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1","Type":"ContainerStarted","Data":"a712d2fdaa7ff39cf1988d065668d30195e43f9806e6684f0c2cf87e6405d81d"} Jul 15 13:34:27.421898 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:27.421860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" event={"ID":"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1","Type":"ContainerStarted","Data":"ee2fba1c0406ed709a67a4abba6941565385608e1a033fb31450185645b6b2a4"} Jul 15 13:34:28.436555 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:28.436517 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1" containerID="ee2fba1c0406ed709a67a4abba6941565385608e1a033fb31450185645b6b2a4" exitCode=0 Jul 15 13:34:28.437020 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:28.436606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" event={"ID":"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1","Type":"ContainerDied","Data":"ee2fba1c0406ed709a67a4abba6941565385608e1a033fb31450185645b6b2a4"} Jul 15 13:34:29.442418 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:29.442366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" event={"ID":"7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1","Type":"ContainerStarted","Data":"29b85f5c79c104ff7adc00f6e71a7d7bd0af1ae2b6466daa677fa5d6199c0970"} Jul 15 13:34:29.442890 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:29.442517 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:29.469509 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:29.469457 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" podStartSLOduration=2.796111132 podStartE2EDuration="7.469439775s" totalImagesPullingTime="3.601165345s" totalInitContainerRuntime="1.072163298s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:34:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:34:29.468927921 +0000 UTC m=+310.986552122" watchObservedRunningTime="2026-07-15 13:34:29.469439775 +0000 UTC m=+310.987063976" Jul 15 13:34:40.449092 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:40.449060 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84978f4d9d-5px9w" Jul 15 13:34:42.422728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.422692 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl"] Jul 15 13:34:42.424784 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.424768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.426883 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.426860 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 13:34:42.427001 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.426946 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 13:34:42.427484 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.427464 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 13:34:42.427577 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.427555 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 13:34:42.427622 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.427593 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:34:42.427622 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.427602 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-492d4\"" Jul 15 13:34:42.442419 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.442387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl"] Jul 15 13:34:42.576537 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.576504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-cert\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.576728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.576548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-manager-config\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.576728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.576653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-metrics-cert\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.576728 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.576703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q8w\" (UniqueName: \"kubernetes.io/projected/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-kube-api-access-h6q8w\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.677945 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.677848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q8w\" (UniqueName: \"kubernetes.io/projected/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-kube-api-access-h6q8w\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.678101 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.677956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-cert\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.678101 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.678009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-manager-config\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.678101 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.678067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-metrics-cert\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.679042 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.678991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-manager-config\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.681121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.681086 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-cert\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.681308 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.681285 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-metrics-cert\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.695001 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.694948 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q8w\" (UniqueName: \"kubernetes.io/projected/c87885ef-a3e9-41f1-b969-3bf7168fb7fb-kube-api-access-h6q8w\") pod \"lws-controller-manager-74d75685f8-vl2jl\" (UID: \"c87885ef-a3e9-41f1-b969-3bf7168fb7fb\") " pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.733797 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.733758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:42.909273 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.909145 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl"] Jul 15 13:34:42.912134 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:34:42.912076 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87885ef_a3e9_41f1_b969_3bf7168fb7fb.slice/crio-d6c6e82e4e8df912f3921d5b99269b9e4743de4dc072199529287ef7c6c85179 WatchSource:0}: Error finding container d6c6e82e4e8df912f3921d5b99269b9e4743de4dc072199529287ef7c6c85179: Status 404 returned error can't find the container with id d6c6e82e4e8df912f3921d5b99269b9e4743de4dc072199529287ef7c6c85179 Jul 15 13:34:42.948263 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.948209 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj"] Jul 15 13:34:42.951025 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.951006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:42.967532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.967506 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 13:34:42.967642 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.967568 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 13:34:42.967642 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.967634 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kjjpt\"" Jul 15 13:34:42.967758 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.967690 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 13:34:42.969560 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.969543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 13:34:42.985667 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:42.985646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj"] Jul 15 13:34:43.081231 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.081202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-tmp\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.081231 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.081236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjh5\" (UniqueName: \"kubernetes.io/projected/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-kube-api-access-8sjh5\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.081460 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.081266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-tls-certs\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.182348 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.182313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-tls-certs\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.182569 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.182429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-tmp\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.182569 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.182463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjh5\" (UniqueName: \"kubernetes.io/projected/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-kube-api-access-8sjh5\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.185198 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.185176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-tmp\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.185410 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.185381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-tls-certs\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.209141 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.209083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjh5\" (UniqueName: \"kubernetes.io/projected/cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4-kube-api-access-8sjh5\") pod \"kube-auth-proxy-9ddcd897f-gkpkj\" (UID: \"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4\") " pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.261048 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.261019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" Jul 15 13:34:43.404243 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.404112 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj"] Jul 15 13:34:43.406792 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:34:43.406769 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbb911c_dd9d_4b00_bc6d_d2afbd3a1df4.slice/crio-78c4378bcaa53fcd6e45ab4d8249cbc557014b7dcdc64b9ea22e8f000f08904b WatchSource:0}: Error finding container 78c4378bcaa53fcd6e45ab4d8249cbc557014b7dcdc64b9ea22e8f000f08904b: Status 404 returned error can't find the container with id 78c4378bcaa53fcd6e45ab4d8249cbc557014b7dcdc64b9ea22e8f000f08904b Jul 15 13:34:43.505242 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.505155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" event={"ID":"c87885ef-a3e9-41f1-b969-3bf7168fb7fb","Type":"ContainerStarted","Data":"d6c6e82e4e8df912f3921d5b99269b9e4743de4dc072199529287ef7c6c85179"} Jul 15 13:34:43.506442 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:43.506418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" event={"ID":"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4","Type":"ContainerStarted","Data":"78c4378bcaa53fcd6e45ab4d8249cbc557014b7dcdc64b9ea22e8f000f08904b"} Jul 15 13:34:46.523126 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:46.523087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" event={"ID":"cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4","Type":"ContainerStarted","Data":"7d53eea2cb5b5ed62bf53d2cafc17ceca02c63fc0133e2a31e51f185a0e5ee84"} Jul 15 13:34:46.524546 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:46.524521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" event={"ID":"c87885ef-a3e9-41f1-b969-3bf7168fb7fb","Type":"ContainerStarted","Data":"f9dba69b6fc47ae9ae57fe92c6828ce05a0ac793ca0e341a441d2c4bece87d90"} Jul 15 13:34:46.524670 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:46.524655 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:34:46.547547 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:46.547503 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9ddcd897f-gkpkj" podStartSLOduration=1.598696848 podStartE2EDuration="4.547489014s" totalImagesPullingTime="2.948792166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:34:46.547016523 +0000 UTC m=+328.064640723" watchObservedRunningTime="2026-07-15 13:34:46.547489014 +0000 UTC m=+328.065113213" Jul 15 13:34:46.579153 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:46.579103 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" podStartSLOduration=2.000555027 podStartE2EDuration="4.579085726s" totalImagesPullingTime="2.578530699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:34:46.577497665 +0000 UTC m=+328.095121865" watchObservedRunningTime="2026-07-15 13:34:46.579085726 +0000 UTC m=+328.096709926" Jul 15 13:34:57.531533 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:34:57.531504 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74d75685f8-vl2jl" Jul 15 13:36:43.551565 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.551492 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq"] Jul 15 13:36:43.554828 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.554811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:43.556867 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.556848 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 13:36:43.556867 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.556858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zsxrl\"" Jul 15 13:36:43.557043 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.557022 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 13:36:43.573182 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.573155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq"] Jul 15 13:36:43.659846 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.659814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvmg\" (UniqueName: \"kubernetes.io/projected/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-kube-api-access-ksvmg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:43.660003 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.659878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:43.760909 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.760867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvmg\" (UniqueName: \"kubernetes.io/projected/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-kube-api-access-ksvmg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:43.761055 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.760965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:43.761328 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.761308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:43.772587 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.772558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvmg\" (UniqueName: \"kubernetes.io/projected/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-kube-api-access-ksvmg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:43.864691 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:43.864656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:44.006032 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:44.006002 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq"] Jul 15 13:36:44.009242 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:36:44.009214 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebc7417_0f0d_4a9b_89ca_f59d4e3423c0.slice/crio-ef520ad93418011ba2dac0c6acf558476a4ec322fd125d56f55b79f7b34666dd WatchSource:0}: Error finding container ef520ad93418011ba2dac0c6acf558476a4ec322fd125d56f55b79f7b34666dd: Status 404 returned error can't find the container with id ef520ad93418011ba2dac0c6acf558476a4ec322fd125d56f55b79f7b34666dd Jul 15 13:36:44.072451 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:44.072408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" event={"ID":"febc7417-0f0d-4a9b-89ca-f59d4e3423c0","Type":"ContainerStarted","Data":"ef520ad93418011ba2dac0c6acf558476a4ec322fd125d56f55b79f7b34666dd"} Jul 15 13:36:51.105808 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:51.105735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" event={"ID":"febc7417-0f0d-4a9b-89ca-f59d4e3423c0","Type":"ContainerStarted","Data":"96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d"} Jul 15 13:36:51.106191 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:51.105871 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:36:51.141417 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:36:51.141354 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" podStartSLOduration=1.584959896 podStartE2EDuration="8.141339079s" totalImagesPullingTime="6.556379183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:36:51.139458579 +0000 UTC m=+452.657082778" watchObservedRunningTime="2026-07-15 13:36:51.141339079 +0000 UTC m=+452.658963279" Jul 15 13:37:02.112309 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:02.112277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:03.922224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.922193 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq"] Jul 15 13:37:03.922695 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.922374 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" containerName="manager" containerID="cri-o://96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d" gracePeriod=2 Jul 15 13:37:03.936869 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.936839 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq"] Jul 15 13:37:03.938669 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:37:03.938632 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-48vdq\": pods \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:03.953760 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.953732 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9"] Jul 15 13:37:03.954094 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.954081 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" containerName="manager" Jul 15 13:37:03.954094 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.954094 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" containerName="manager" Jul 15 13:37:03.954213 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.954159 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" containerName="manager" Jul 15 13:37:03.957143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.957122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:03.958886 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:37:03.958855 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:03.975170 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:03.975138 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9"] Jul 15 13:37:04.049015 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.048986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v25j\" (UniqueName: \"kubernetes.io/projected/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-kube-api-access-5v25j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jw5j9\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:04.049125 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.049045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jw5j9\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:04.150514 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.150454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jw5j9\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:04.150649 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.150635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v25j\" (UniqueName: \"kubernetes.io/projected/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-kube-api-access-5v25j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jw5j9\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:04.150732 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.150713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jw5j9\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:04.160699 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.160675 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:04.162504 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:37:04.162477 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:04.162981 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.162957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v25j\" (UniqueName: \"kubernetes.io/projected/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-kube-api-access-5v25j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jw5j9\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:04.167647 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.167615 2576 generic.go:356] "Generic (PLEG): container finished" podID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" containerID="96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d" exitCode=0 Jul 15 13:37:04.167748 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.167662 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:04.167748 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.167685 2576 scope.go:117] "RemoveContainer" containerID="96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d" Jul 15 13:37:04.169481 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:37:04.169457 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:04.175939 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.175924 2576 scope.go:117] "RemoveContainer" containerID="96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d" Jul 15 13:37:04.176177 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:37:04.176161 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d\": container with ID starting with 96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d not found: ID does not exist" containerID="96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d" Jul 15 13:37:04.176240 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.176183 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d"} err="failed to get container status \"96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d\": rpc error: code = NotFound desc = could not find container \"96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d\": container with ID starting with 96707fb7708d8776a4b5b43ff3992bdf373e592d621a9e1168c9eef092f7ed3d not found: ID does not exist" Jul 15 13:37:04.251942 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.251903 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksvmg\" (UniqueName: \"kubernetes.io/projected/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-kube-api-access-ksvmg\") pod \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " Jul 15 13:37:04.252098 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.252012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-extensions-socket-volume\") pod \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\" (UID: \"febc7417-0f0d-4a9b-89ca-f59d4e3423c0\") " Jul 15 13:37:04.252543 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.252511 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "febc7417-0f0d-4a9b-89ca-f59d4e3423c0" (UID: "febc7417-0f0d-4a9b-89ca-f59d4e3423c0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:37:04.253979 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.253960 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-kube-api-access-ksvmg" (OuterVolumeSpecName: "kube-api-access-ksvmg") pod "febc7417-0f0d-4a9b-89ca-f59d4e3423c0" (UID: "febc7417-0f0d-4a9b-89ca-f59d4e3423c0"). InnerVolumeSpecName "kube-api-access-ksvmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:37:04.321183 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.321142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:04.353504 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.353473 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-extensions-socket-volume\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:37:04.353504 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.353500 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvmg\" (UniqueName: \"kubernetes.io/projected/febc7417-0f0d-4a9b-89ca-f59d4e3423c0-kube-api-access-ksvmg\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:37:04.464438 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:04.464346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9"] Jul 15 13:37:04.467984 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:37:04.467954 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b2cfcb_6015_4705_8f87_a3bd3f566bb0.slice/crio-f658e17d8ff50de5a69fa885a54ee17598e871a1079427b2175c5ff52ae6f75c WatchSource:0}: Error finding container f658e17d8ff50de5a69fa885a54ee17598e871a1079427b2175c5ff52ae6f75c: Status 404 returned error can't find the container with id f658e17d8ff50de5a69fa885a54ee17598e871a1079427b2175c5ff52ae6f75c Jul 15 13:37:04.478585 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:37:04.478554 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-48vdq\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-48vdq" Jul 15 13:37:05.096486 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:05.096447 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="febc7417-0f0d-4a9b-89ca-f59d4e3423c0" path="/var/lib/kubelet/pods/febc7417-0f0d-4a9b-89ca-f59d4e3423c0/volumes" Jul 15 13:37:05.174022 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:05.173986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" event={"ID":"06b2cfcb-6015-4705-8f87-a3bd3f566bb0","Type":"ContainerStarted","Data":"1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4"} Jul 15 13:37:05.174223 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:05.174045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" event={"ID":"06b2cfcb-6015-4705-8f87-a3bd3f566bb0","Type":"ContainerStarted","Data":"f658e17d8ff50de5a69fa885a54ee17598e871a1079427b2175c5ff52ae6f75c"} Jul 15 13:37:05.174223 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:05.174163 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:05.223260 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:05.223198 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" podStartSLOduration=2.223174676 podStartE2EDuration="2.223174676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:37:05.222609427 +0000 UTC m=+466.740233628" watchObservedRunningTime="2026-07-15 13:37:05.223174676 +0000 UTC m=+466.740798876" Jul 15 13:37:16.181710 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:16.181681 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:20.199580 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.199546 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9"] Jul 15 13:37:20.199975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.199752 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" podUID="06b2cfcb-6015-4705-8f87-a3bd3f566bb0" containerName="manager" containerID="cri-o://1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4" gracePeriod=10 Jul 15 13:37:20.453116 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.453060 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:20.601329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.601296 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-extensions-socket-volume\") pod \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " Jul 15 13:37:20.601510 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.601383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v25j\" (UniqueName: \"kubernetes.io/projected/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-kube-api-access-5v25j\") pod \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\" (UID: \"06b2cfcb-6015-4705-8f87-a3bd3f566bb0\") " Jul 15 13:37:20.601679 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.601655 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "06b2cfcb-6015-4705-8f87-a3bd3f566bb0" (UID: "06b2cfcb-6015-4705-8f87-a3bd3f566bb0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:37:20.603356 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.603331 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-kube-api-access-5v25j" (OuterVolumeSpecName: "kube-api-access-5v25j") pod "06b2cfcb-6015-4705-8f87-a3bd3f566bb0" (UID: "06b2cfcb-6015-4705-8f87-a3bd3f566bb0"). InnerVolumeSpecName "kube-api-access-5v25j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:37:20.702493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.702464 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v25j\" (UniqueName: \"kubernetes.io/projected/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-kube-api-access-5v25j\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:37:20.702493 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:20.702488 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06b2cfcb-6015-4705-8f87-a3bd3f566bb0-extensions-socket-volume\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:37:21.249200 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.249168 2576 generic.go:356] "Generic (PLEG): container finished" podID="06b2cfcb-6015-4705-8f87-a3bd3f566bb0" containerID="1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4" exitCode=0 Jul 15 13:37:21.249601 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.249215 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" Jul 15 13:37:21.249601 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.249229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" event={"ID":"06b2cfcb-6015-4705-8f87-a3bd3f566bb0","Type":"ContainerDied","Data":"1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4"} Jul 15 13:37:21.249601 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.249298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9" event={"ID":"06b2cfcb-6015-4705-8f87-a3bd3f566bb0","Type":"ContainerDied","Data":"f658e17d8ff50de5a69fa885a54ee17598e871a1079427b2175c5ff52ae6f75c"} Jul 15 13:37:21.249601 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.249329 2576 scope.go:117] "RemoveContainer" containerID="1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4" Jul 15 13:37:21.257557 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.257539 2576 scope.go:117] "RemoveContainer" containerID="1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4" Jul 15 13:37:21.257803 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:37:21.257777 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4\": container with ID starting with 1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4 not found: ID does not exist" containerID="1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4" Jul 15 13:37:21.257844 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.257812 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4"} err="failed to get container status \"1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4\": rpc error: code = NotFound desc = could not find container \"1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4\": container with ID starting with 1ee9f1b3de97daf828ca8ba790147b3e83ec2308f247df72e543110a83544aa4 not found: ID does not exist" Jul 15 13:37:21.270940 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.270912 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9"] Jul 15 13:37:21.278322 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:21.278294 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jw5j9"] Jul 15 13:37:23.096169 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:23.096138 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b2cfcb-6015-4705-8f87-a3bd3f566bb0" path="/var/lib/kubelet/pods/06b2cfcb-6015-4705-8f87-a3bd3f566bb0/volumes" Jul 15 13:37:36.596991 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.596955 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m"] Jul 15 13:37:36.597464 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.597373 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b2cfcb-6015-4705-8f87-a3bd3f566bb0" containerName="manager" Jul 15 13:37:36.597464 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.597390 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b2cfcb-6015-4705-8f87-a3bd3f566bb0" containerName="manager" Jul 15 13:37:36.597600 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.597511 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b2cfcb-6015-4705-8f87-a3bd3f566bb0" containerName="manager" Jul 15 13:37:36.601854 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.601834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.604025 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.604007 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5hmb5\"" Jul 15 13:37:36.604480 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.604466 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 13:37:36.614845 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.614828 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m"] Jul 15 13:37:36.742299 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742299 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742509 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742509 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105d60bd-9431-4f83-98a2-1948172c7c5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742509 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5475\" (UniqueName: \"kubernetes.io/projected/105d60bd-9431-4f83-98a2-1948172c7c5c-kube-api-access-s5475\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742509 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742648 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742648 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.742648 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.742595 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843446 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843446 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843657 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843657 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105d60bd-9431-4f83-98a2-1948172c7c5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843657 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5475\" (UniqueName: \"kubernetes.io/projected/105d60bd-9431-4f83-98a2-1948172c7c5c-kube-api-access-s5475\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843807 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843807 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843807 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.843935 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.843830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.844081 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.844058 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.844242 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.844213 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.844351 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.844275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.844351 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.844339 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105d60bd-9431-4f83-98a2-1948172c7c5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.844494 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.844473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.845863 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.845844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.845950 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.845935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.852268 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.852199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5475\" (UniqueName: \"kubernetes.io/projected/105d60bd-9431-4f83-98a2-1948172c7c5c-kube-api-access-s5475\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.854197 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.854178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105d60bd-9431-4f83-98a2-1948172c7c5c-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nv52m\" (UID: \"105d60bd-9431-4f83-98a2-1948172c7c5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:36.911224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:36.911192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:37.050166 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:37.050129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m"] Jul 15 13:37:37.053242 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:37:37.053213 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105d60bd_9431_4f83_98a2_1948172c7c5c.slice/crio-b159085a6e291901fcd16efb8fb7d21b195eef23e120eb80819b4e079b0671a4 WatchSource:0}: Error finding container b159085a6e291901fcd16efb8fb7d21b195eef23e120eb80819b4e079b0671a4: Status 404 returned error can't find the container with id b159085a6e291901fcd16efb8fb7d21b195eef23e120eb80819b4e079b0671a4 Jul 15 13:37:37.321714 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:37.321622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" event={"ID":"105d60bd-9431-4f83-98a2-1948172c7c5c","Type":"ContainerStarted","Data":"b159085a6e291901fcd16efb8fb7d21b195eef23e120eb80819b4e079b0671a4"} Jul 15 13:37:39.423212 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:39.423171 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:37:39.423497 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:39.423240 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:37:39.423497 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:39.423270 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:37:40.338813 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:40.338776 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" event={"ID":"105d60bd-9431-4f83-98a2-1948172c7c5c","Type":"ContainerStarted","Data":"66939d052bbbf636341dc79b9f50102e1019e307ae3bc28ca66d52e1df974101"} Jul 15 13:37:40.361925 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:40.361879 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" podStartSLOduration=1.994227248 podStartE2EDuration="4.361864715s" totalImagesPullingTime="2.367637467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:37:40.359847404 +0000 UTC m=+501.877471603" watchObservedRunningTime="2026-07-15 13:37:40.361864715 +0000 UTC m=+501.879488914" Jul 15 13:37:40.911738 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:40.911706 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:40.916500 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:40.916477 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:41.343185 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:41.343095 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:41.344090 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:41.344071 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:37:53.685255 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.685217 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:37:53.688800 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.688777 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:53.691189 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.691166 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 13:37:53.691289 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.691166 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 13:37:53.691289 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.691211 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 13:37:53.691767 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.691753 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wd5w4\"" Jul 15 13:37:53.700657 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.700635 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:37:53.715372 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.715350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:37:53.799795 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.799763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbvcx\" (UniqueName: \"kubernetes.io/projected/d56ad4c7-45a8-4626-9100-afec43bb2d38-kube-api-access-tbvcx\") pod \"limitador-limitador-54d6d988d5-g8228\" (UID: \"d56ad4c7-45a8-4626-9100-afec43bb2d38\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:53.799913 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.799826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d56ad4c7-45a8-4626-9100-afec43bb2d38-config-file\") pod \"limitador-limitador-54d6d988d5-g8228\" (UID: \"d56ad4c7-45a8-4626-9100-afec43bb2d38\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:53.900863 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.900833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbvcx\" (UniqueName: \"kubernetes.io/projected/d56ad4c7-45a8-4626-9100-afec43bb2d38-kube-api-access-tbvcx\") pod \"limitador-limitador-54d6d988d5-g8228\" (UID: \"d56ad4c7-45a8-4626-9100-afec43bb2d38\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:53.901047 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.900889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d56ad4c7-45a8-4626-9100-afec43bb2d38-config-file\") pod \"limitador-limitador-54d6d988d5-g8228\" (UID: \"d56ad4c7-45a8-4626-9100-afec43bb2d38\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:53.901557 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.901541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d56ad4c7-45a8-4626-9100-afec43bb2d38-config-file\") pod \"limitador-limitador-54d6d988d5-g8228\" (UID: \"d56ad4c7-45a8-4626-9100-afec43bb2d38\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:53.912809 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.912783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbvcx\" (UniqueName: \"kubernetes.io/projected/d56ad4c7-45a8-4626-9100-afec43bb2d38-kube-api-access-tbvcx\") pod \"limitador-limitador-54d6d988d5-g8228\" (UID: \"d56ad4c7-45a8-4626-9100-afec43bb2d38\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:53.999233 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:53.999168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:54.124599 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.124571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:37:54.126775 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:37:54.126753 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56ad4c7_45a8_4626_9100_afec43bb2d38.slice/crio-f0fa54025251a8650af86a8d3d7186af66dc9e38832536f1efa65e0854ac3376 WatchSource:0}: Error finding container f0fa54025251a8650af86a8d3d7186af66dc9e38832536f1efa65e0854ac3376: Status 404 returned error can't find the container with id f0fa54025251a8650af86a8d3d7186af66dc9e38832536f1efa65e0854ac3376 Jul 15 13:37:54.304150 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.304075 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-9pzl6"] Jul 15 13:37:54.308833 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.308817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9pzl6" Jul 15 13:37:54.310709 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.310688 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p2ldb\"" Jul 15 13:37:54.316895 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.316872 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9pzl6"] Jul 15 13:37:54.403966 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.403931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdxj\" (UniqueName: \"kubernetes.io/projected/2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8-kube-api-access-lcdxj\") pod \"authorino-75d556bd77-9pzl6\" (UID: \"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8\") " pod="kuadrant-system/authorino-75d556bd77-9pzl6" Jul 15 13:37:54.408444 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.408416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" event={"ID":"d56ad4c7-45a8-4626-9100-afec43bb2d38","Type":"ContainerStarted","Data":"f0fa54025251a8650af86a8d3d7186af66dc9e38832536f1efa65e0854ac3376"} Jul 15 13:37:54.505234 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.505211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdxj\" (UniqueName: \"kubernetes.io/projected/2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8-kube-api-access-lcdxj\") pod \"authorino-75d556bd77-9pzl6\" (UID: \"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8\") " pod="kuadrant-system/authorino-75d556bd77-9pzl6" Jul 15 13:37:54.514336 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.514310 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdxj\" (UniqueName: \"kubernetes.io/projected/2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8-kube-api-access-lcdxj\") pod \"authorino-75d556bd77-9pzl6\" (UID: \"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8\") " pod="kuadrant-system/authorino-75d556bd77-9pzl6" Jul 15 13:37:54.618539 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.618514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9pzl6" Jul 15 13:37:54.741389 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:54.741360 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9pzl6"] Jul 15 13:37:54.744113 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:37:54.744083 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dcbdb1c_d6e6_4b45_a840_bfe5295b83f8.slice/crio-1d73939e3383ace261f0f5ca3c07c5d991c54cc1ab9cd58755a18a28fa26e2cb WatchSource:0}: Error finding container 1d73939e3383ace261f0f5ca3c07c5d991c54cc1ab9cd58755a18a28fa26e2cb: Status 404 returned error can't find the container with id 1d73939e3383ace261f0f5ca3c07c5d991c54cc1ab9cd58755a18a28fa26e2cb Jul 15 13:37:55.417196 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:55.417114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9pzl6" event={"ID":"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8","Type":"ContainerStarted","Data":"1d73939e3383ace261f0f5ca3c07c5d991c54cc1ab9cd58755a18a28fa26e2cb"} Jul 15 13:37:59.439919 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:59.439879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9pzl6" event={"ID":"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8","Type":"ContainerStarted","Data":"c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1"} Jul 15 13:37:59.441364 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:59.441343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" event={"ID":"d56ad4c7-45a8-4626-9100-afec43bb2d38","Type":"ContainerStarted","Data":"c4f46166290d2cc5795ed67e7ad0e6f7dfb709cadcae0ae3601180b1c00f0b59"} Jul 15 13:37:59.441475 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:59.441427 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:37:59.464292 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:59.464219 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-9pzl6" podStartSLOduration=1.40205657 podStartE2EDuration="5.464207159s" totalImagesPullingTime="4.062150589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:37:59.462594116 +0000 UTC m=+520.980218315" watchObservedRunningTime="2026-07-15 13:37:59.464207159 +0000 UTC m=+520.981831356" Jul 15 13:37:59.490058 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:37:59.490006 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" podStartSLOduration=1.812002809 podStartE2EDuration="6.489988198s" totalImagesPullingTime="4.677985389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:37:59.487603094 +0000 UTC m=+521.005227294" watchObservedRunningTime="2026-07-15 13:37:59.489988198 +0000 UTC m=+521.007612398" Jul 15 13:38:10.445988 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:10.445959 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-g8228" Jul 15 13:38:38.256102 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.256069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67f965f66f-rxrwv"] Jul 15 13:38:38.264648 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.264626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.266890 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.266870 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 13:38:38.272862 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.272837 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67f965f66f-rxrwv"] Jul 15 13:38:38.305074 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.305048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfn4d\" (UniqueName: \"kubernetes.io/projected/49319879-5801-4b14-8b4d-a1bcf920c09b-kube-api-access-nfn4d\") pod \"authorino-67f965f66f-rxrwv\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.305188 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.305081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49319879-5801-4b14-8b4d-a1bcf920c09b-tls-cert\") pod \"authorino-67f965f66f-rxrwv\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.405564 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.405522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfn4d\" (UniqueName: \"kubernetes.io/projected/49319879-5801-4b14-8b4d-a1bcf920c09b-kube-api-access-nfn4d\") pod \"authorino-67f965f66f-rxrwv\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.405753 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.405578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49319879-5801-4b14-8b4d-a1bcf920c09b-tls-cert\") pod \"authorino-67f965f66f-rxrwv\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.407926 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.407903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49319879-5801-4b14-8b4d-a1bcf920c09b-tls-cert\") pod \"authorino-67f965f66f-rxrwv\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.414152 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.414125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfn4d\" (UniqueName: \"kubernetes.io/projected/49319879-5801-4b14-8b4d-a1bcf920c09b-kube-api-access-nfn4d\") pod \"authorino-67f965f66f-rxrwv\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.574505 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.574422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:38:38.702638 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:38.702613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67f965f66f-rxrwv"] Jul 15 13:38:38.705121 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:38:38.705091 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49319879_5801_4b14_8b4d_a1bcf920c09b.slice/crio-a78f7ab12a364b8b2a4f95272ab2bcdf57f9b43174c6a0e024d2304b2f2a872f WatchSource:0}: Error finding container a78f7ab12a364b8b2a4f95272ab2bcdf57f9b43174c6a0e024d2304b2f2a872f: Status 404 returned error can't find the container with id a78f7ab12a364b8b2a4f95272ab2bcdf57f9b43174c6a0e024d2304b2f2a872f Jul 15 13:38:39.636976 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:39.636938 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67f965f66f-rxrwv" event={"ID":"49319879-5801-4b14-8b4d-a1bcf920c09b","Type":"ContainerStarted","Data":"aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34"} Jul 15 13:38:39.637362 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:39.636984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67f965f66f-rxrwv" event={"ID":"49319879-5801-4b14-8b4d-a1bcf920c09b","Type":"ContainerStarted","Data":"a78f7ab12a364b8b2a4f95272ab2bcdf57f9b43174c6a0e024d2304b2f2a872f"} Jul 15 13:38:39.659092 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:39.659043 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67f965f66f-rxrwv" podStartSLOduration=1.251789921 podStartE2EDuration="1.659027707s" totalImagesPullingTime="407.237786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:38:39.65821117 +0000 UTC m=+561.175835372" watchObservedRunningTime="2026-07-15 13:38:39.659027707 +0000 UTC m=+561.176651906" Jul 15 13:38:39.691015 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:39.690983 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9pzl6"] Jul 15 13:38:39.691204 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:39.691181 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-9pzl6" podUID="2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8" containerName="authorino" containerID="cri-o://c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1" gracePeriod=30 Jul 15 13:38:39.941674 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:39.941649 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9pzl6" Jul 15 13:38:40.022164 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.022125 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcdxj\" (UniqueName: \"kubernetes.io/projected/2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8-kube-api-access-lcdxj\") pod \"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8\" (UID: \"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8\") " Jul 15 13:38:40.024182 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.024147 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8-kube-api-access-lcdxj" (OuterVolumeSpecName: "kube-api-access-lcdxj") pod "2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8" (UID: "2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8"). InnerVolumeSpecName "kube-api-access-lcdxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:38:40.122889 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.122852 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcdxj\" (UniqueName: \"kubernetes.io/projected/2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8-kube-api-access-lcdxj\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:38:40.642853 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.642815 2576 generic.go:356] "Generic (PLEG): container finished" podID="2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8" containerID="c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1" exitCode=0 Jul 15 13:38:40.643311 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.642862 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9pzl6" Jul 15 13:38:40.643311 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.642907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9pzl6" event={"ID":"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8","Type":"ContainerDied","Data":"c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1"} Jul 15 13:38:40.643311 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.642962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9pzl6" event={"ID":"2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8","Type":"ContainerDied","Data":"1d73939e3383ace261f0f5ca3c07c5d991c54cc1ab9cd58755a18a28fa26e2cb"} Jul 15 13:38:40.643311 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.642998 2576 scope.go:117] "RemoveContainer" containerID="c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1" Jul 15 13:38:40.651734 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.651718 2576 scope.go:117] "RemoveContainer" containerID="c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1" Jul 15 13:38:40.651995 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:38:40.651977 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1\": container with ID starting with c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1 not found: ID does not exist" containerID="c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1" Jul 15 13:38:40.652039 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.652005 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1"} err="failed to get container status \"c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1\": rpc error: code = NotFound desc = could not find container \"c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1\": container with ID starting with c12ca998f5153339a81ffcfc68780bee92c920308f898685e687694df6fc2ee1 not found: ID does not exist" Jul 15 13:38:40.671436 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.671411 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9pzl6"] Jul 15 13:38:40.678066 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:40.678046 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9pzl6"] Jul 15 13:38:41.096480 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:38:41.096453 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8" path="/var/lib/kubelet/pods/2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8/volumes" Jul 15 13:39:18.895544 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:18.895515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:39:18.898157 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:18.898139 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:39:22.573922 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:22.573888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:39:30.881080 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:30.881039 2576 generic.go:356] "Generic (PLEG): container finished" podID="105d60bd-9431-4f83-98a2-1948172c7c5c" containerID="66939d052bbbf636341dc79b9f50102e1019e307ae3bc28ca66d52e1df974101" exitCode=0 Jul 15 13:39:30.881511 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:30.881112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" event={"ID":"105d60bd-9431-4f83-98a2-1948172c7c5c","Type":"ContainerDied","Data":"66939d052bbbf636341dc79b9f50102e1019e307ae3bc28ca66d52e1df974101"} Jul 15 13:39:30.881554 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:30.881509 2576 scope.go:117] "RemoveContainer" containerID="66939d052bbbf636341dc79b9f50102e1019e307ae3bc28ca66d52e1df974101" Jul 15 13:39:30.882476 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:30.882449 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:39:30.882560 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:30.882516 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:39:30.882560 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:30.882545 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:39:31.887288 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:31.887253 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" event={"ID":"105d60bd-9431-4f83-98a2-1948172c7c5c","Type":"ContainerStarted","Data":"81061f4ffdfba77be7b33f135655abd42280eb381f9dfc8737f4e39de8a0b4e5"} Jul 15 13:39:31.911477 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:31.911446 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:39:31.911704 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:31.911683 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 15 13:39:31.911772 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:31.911726 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" podUID="105d60bd-9431-4f83-98a2-1948172c7c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 15 13:39:32.911771 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:32.911737 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 15 13:39:32.912143 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:32.911796 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" podUID="105d60bd-9431-4f83-98a2-1948172c7c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 15 13:39:33.912413 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:33.912371 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 15 13:39:33.912885 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:33.912461 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" podUID="105d60bd-9431-4f83-98a2-1948172c7c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 15 13:39:34.912494 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:34.912459 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nv52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 15 13:39:34.912858 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:34.912514 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" podUID="105d60bd-9431-4f83-98a2-1948172c7c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 15 13:39:35.915605 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:35.915574 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:39:35.916070 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:35.915903 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:39:35.916754 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:35.916734 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nv52m" Jul 15 13:39:40.120685 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:40.120649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:39:48.488860 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:48.488770 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:39:54.184734 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:39:54.184699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:40:04.378987 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.378951 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949"] Jul 15 13:40:04.379486 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.379316 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8" containerName="authorino" Jul 15 13:40:04.379486 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.379327 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8" containerName="authorino" Jul 15 13:40:04.379486 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.379385 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcbdb1c-d6e6-4b45-a840-bfe5295b83f8" containerName="authorino" Jul 15 13:40:04.382633 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.382617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.385360 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.385339 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 13:40:04.385479 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.385374 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 13:40:04.385520 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.385499 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bdn7z\"" Jul 15 13:40:04.385651 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.385637 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 15 13:40:04.398154 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.398126 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949"] Jul 15 13:40:04.504991 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.504955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448c25a8-9d10-44d8-87bf-c0b22818cc28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.505160 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.504997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.505160 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.505103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.505160 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.505128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.505269 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.505162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbjd\" (UniqueName: \"kubernetes.io/projected/448c25a8-9d10-44d8-87bf-c0b22818cc28-kube-api-access-rqbjd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.505269 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.505233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606245 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448c25a8-9d10-44d8-87bf-c0b22818cc28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606245 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606525 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606525 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606525 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbjd\" (UniqueName: \"kubernetes.io/projected/448c25a8-9d10-44d8-87bf-c0b22818cc28-kube-api-access-rqbjd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606525 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606806 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606880 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.606880 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.606841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.608636 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.608613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448c25a8-9d10-44d8-87bf-c0b22818cc28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.608797 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.608779 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448c25a8-9d10-44d8-87bf-c0b22818cc28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.615450 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.615423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbjd\" (UniqueName: \"kubernetes.io/projected/448c25a8-9d10-44d8-87bf-c0b22818cc28-kube-api-access-rqbjd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949\" (UID: \"448c25a8-9d10-44d8-87bf-c0b22818cc28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.693848 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.693753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:04.828942 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.828895 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949"] Jul 15 13:40:04.832470 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:40:04.832441 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448c25a8_9d10_44d8_87bf_c0b22818cc28.slice/crio-6cd10a0a1cb0d9ead11fd282ac2b20c05c8f7d611cd399db6b6b80a92bf88c19 WatchSource:0}: Error finding container 6cd10a0a1cb0d9ead11fd282ac2b20c05c8f7d611cd399db6b6b80a92bf88c19: Status 404 returned error can't find the container with id 6cd10a0a1cb0d9ead11fd282ac2b20c05c8f7d611cd399db6b6b80a92bf88c19 Jul 15 13:40:04.833882 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:04.833863 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:40:05.055706 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:05.055623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" event={"ID":"448c25a8-9d10-44d8-87bf-c0b22818cc28","Type":"ContainerStarted","Data":"6cd10a0a1cb0d9ead11fd282ac2b20c05c8f7d611cd399db6b6b80a92bf88c19"} Jul 15 13:40:06.096112 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:06.096087 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:40:07.067277 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:07.067243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" event={"ID":"448c25a8-9d10-44d8-87bf-c0b22818cc28","Type":"ContainerStarted","Data":"b298c3a30fd7b05bdd15c83a4f0c69b002775d12eaa9a776d9f58b8f6e0b520c"} Jul 15 13:40:07.087543 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:07.087489 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" podStartSLOduration=1.8431354629999999 podStartE2EDuration="3.087470972s" totalImagesPullingTime="1.244335509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:40:07.087366348 +0000 UTC m=+648.604990548" watchObservedRunningTime="2026-07-15 13:40:07.087470972 +0000 UTC m=+648.605095192" Jul 15 13:40:14.694202 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:14.694162 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:14.694625 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:14.694226 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:14.706226 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:14.706198 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:15.117106 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:15.117077 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949" Jul 15 13:40:21.280853 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:21.280819 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:40:30.271216 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.271186 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85"] Jul 15 13:40:30.274606 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.274589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.276612 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.276589 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 13:40:30.284976 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.284956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85"] Jul 15 13:40:30.344337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.344310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.344337 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.344337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca01a4d6-2232-4036-a01d-3cd50eb1523b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.344544 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.344356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.344544 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.344375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqwx\" (UniqueName: \"kubernetes.io/projected/ca01a4d6-2232-4036-a01d-3cd50eb1523b-kube-api-access-lzqwx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.344658 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.344537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.344658 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.344596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446060 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446253 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca01a4d6-2232-4036-a01d-3cd50eb1523b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446253 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446253 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqwx\" (UniqueName: \"kubernetes.io/projected/ca01a4d6-2232-4036-a01d-3cd50eb1523b-kube-api-access-lzqwx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446253 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446253 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446619 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446619 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.446714 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.446677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.448291 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.448268 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca01a4d6-2232-4036-a01d-3cd50eb1523b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.448588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.448568 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca01a4d6-2232-4036-a01d-3cd50eb1523b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.454145 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.454126 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqwx\" (UniqueName: \"kubernetes.io/projected/ca01a4d6-2232-4036-a01d-3cd50eb1523b-kube-api-access-lzqwx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85\" (UID: \"ca01a4d6-2232-4036-a01d-3cd50eb1523b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.583128 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.583030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:30.713533 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:30.713510 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85"] Jul 15 13:40:30.715985 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:40:30.715960 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca01a4d6_2232_4036_a01d_3cd50eb1523b.slice/crio-9197948257f9d0cd33c4301fca79f98fde5fe1ace03db5026079eb0f2a3f9b6c WatchSource:0}: Error finding container 9197948257f9d0cd33c4301fca79f98fde5fe1ace03db5026079eb0f2a3f9b6c: Status 404 returned error can't find the container with id 9197948257f9d0cd33c4301fca79f98fde5fe1ace03db5026079eb0f2a3f9b6c Jul 15 13:40:31.183929 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:31.183896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" event={"ID":"ca01a4d6-2232-4036-a01d-3cd50eb1523b","Type":"ContainerStarted","Data":"d9311891f5b5c60b681962fe0360fa25e4c15677b18577e2976f19739c0c4f04"} Jul 15 13:40:31.184147 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:31.183938 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" event={"ID":"ca01a4d6-2232-4036-a01d-3cd50eb1523b","Type":"ContainerStarted","Data":"9197948257f9d0cd33c4301fca79f98fde5fe1ace03db5026079eb0f2a3f9b6c"} Jul 15 13:40:31.212275 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:31.212189 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" podStartSLOduration=0.986915966 podStartE2EDuration="1.21217606s" totalImagesPullingTime="225.260094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:40:31.210770825 +0000 UTC m=+672.728395023" watchObservedRunningTime="2026-07-15 13:40:31.21217606 +0000 UTC m=+672.729800258" Jul 15 13:40:32.994668 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:32.994631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:40:40.583342 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:40.583295 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:40.583342 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:40.583360 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:40.596059 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:40.596036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:40:41.242057 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:40:41.242026 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85" Jul 15 13:41:46.534046 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:46.533964 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67f965f66f-rxrwv"] Jul 15 13:41:46.534667 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:46.534145 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-67f965f66f-rxrwv" podUID="49319879-5801-4b14-8b4d-a1bcf920c09b" containerName="authorino" containerID="cri-o://aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34" gracePeriod=30 Jul 15 13:41:46.794388 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:46.794321 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:41:46.932821 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:46.932770 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfn4d\" (UniqueName: \"kubernetes.io/projected/49319879-5801-4b14-8b4d-a1bcf920c09b-kube-api-access-nfn4d\") pod \"49319879-5801-4b14-8b4d-a1bcf920c09b\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " Jul 15 13:41:46.933026 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:46.932891 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49319879-5801-4b14-8b4d-a1bcf920c09b-tls-cert\") pod \"49319879-5801-4b14-8b4d-a1bcf920c09b\" (UID: \"49319879-5801-4b14-8b4d-a1bcf920c09b\") " Jul 15 13:41:46.934930 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:46.934904 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49319879-5801-4b14-8b4d-a1bcf920c09b-kube-api-access-nfn4d" (OuterVolumeSpecName: "kube-api-access-nfn4d") pod "49319879-5801-4b14-8b4d-a1bcf920c09b" (UID: "49319879-5801-4b14-8b4d-a1bcf920c09b"). InnerVolumeSpecName "kube-api-access-nfn4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:41:46.945618 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:46.945576 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49319879-5801-4b14-8b4d-a1bcf920c09b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "49319879-5801-4b14-8b4d-a1bcf920c09b" (UID: "49319879-5801-4b14-8b4d-a1bcf920c09b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:41:47.033877 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.033830 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfn4d\" (UniqueName: \"kubernetes.io/projected/49319879-5801-4b14-8b4d-a1bcf920c09b-kube-api-access-nfn4d\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:41:47.033877 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.033873 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/49319879-5801-4b14-8b4d-a1bcf920c09b-tls-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:41:47.552412 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.552302 2576 generic.go:356] "Generic (PLEG): container finished" podID="49319879-5801-4b14-8b4d-a1bcf920c09b" containerID="aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34" exitCode=0 Jul 15 13:41:47.552412 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.552345 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67f965f66f-rxrwv" Jul 15 13:41:47.552862 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.552388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67f965f66f-rxrwv" event={"ID":"49319879-5801-4b14-8b4d-a1bcf920c09b","Type":"ContainerDied","Data":"aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34"} Jul 15 13:41:47.552862 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.552458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67f965f66f-rxrwv" event={"ID":"49319879-5801-4b14-8b4d-a1bcf920c09b","Type":"ContainerDied","Data":"a78f7ab12a364b8b2a4f95272ab2bcdf57f9b43174c6a0e024d2304b2f2a872f"} Jul 15 13:41:47.552862 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.552483 2576 scope.go:117] "RemoveContainer" containerID="aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34" Jul 15 13:41:47.561496 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.561476 2576 scope.go:117] "RemoveContainer" containerID="aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34" Jul 15 13:41:47.561804 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:41:47.561777 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34\": container with ID starting with aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34 not found: ID does not exist" containerID="aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34" Jul 15 13:41:47.561892 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.561815 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34"} err="failed to get container status \"aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34\": rpc error: code = NotFound desc = could not find container \"aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34\": container with ID starting with aefdb023dd5e94902b4260cda197fe3275d7150e293bd2ea3db506825c4c0b34 not found: ID does not exist" Jul 15 13:41:47.578859 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.578830 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67f965f66f-rxrwv"] Jul 15 13:41:47.579038 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:47.579026 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-67f965f66f-rxrwv"] Jul 15 13:41:49.096643 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:41:49.096596 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49319879-5801-4b14-8b4d-a1bcf920c09b" path="/var/lib/kubelet/pods/49319879-5801-4b14-8b4d-a1bcf920c09b/volumes" Jul 15 13:42:27.486614 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:42:27.486578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:42:36.974343 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:42:36.974311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:42:48.479546 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:42:48.479465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:42:56.305894 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:42:56.305858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:43:08.390234 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:43:08.390200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:43:35.976179 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:43:35.976145 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:43:48.874349 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:43:48.874309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:44:18.927319 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:44:18.927237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:44:18.929716 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:44:18.929694 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:44:26.769706 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:44:26.769661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:44:35.674897 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:44:35.674862 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:44:45.772464 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:44:45.772429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:44:58.275945 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:44:58.275907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:45:09.674239 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:45:09.674199 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:45:19.400906 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:45:19.400864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:45:30.481176 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:45:30.481141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:45:40.284344 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:45:40.284308 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:46:05.877083 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:46:05.877011 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:46:17.181121 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:46:17.181083 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:46:41.003882 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:46:41.003846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:46:52.176232 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:46:52.176191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:47:06.983837 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:47:06.983805 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:47:16.701264 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:47:16.701184 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:47:26.775506 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:47:26.775472 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:47:36.878392 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:47:36.878354 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:47:51.374456 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:47:51.374418 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:49:18.962356 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:49:18.962322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:49:18.988842 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:49:18.988811 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:49:35.673141 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:49:35.673108 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:49:47.385329 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:49:47.385294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:49:56.573084 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:49:56.573047 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:50:07.825011 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:50:07.824976 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:50:19.279130 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:50:19.279055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:50:30.483627 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:50:30.483586 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:50:38.280680 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:50:38.280636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:50:49.882254 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:50:49.882219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:51:00.780323 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:51:00.780287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:51:09.389824 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:51:09.389781 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:51:20.576155 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:51:20.576113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:51:45.474562 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:51:45.474484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:51:53.376808 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:51:53.376767 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:52:03.474908 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:52:03.474877 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:52:11.970699 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:52:11.970661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:52:22.167962 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:52:22.167927 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:52:50.880326 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:52:50.880289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:53:02.380411 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:53:02.380363 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:53:12.080049 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:53:12.079959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:53:26.374954 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:53:26.374915 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:53:33.868806 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:53:33.868771 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:53:43.987633 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:53:43.987599 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:53:53.189906 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:53:53.189872 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:54:08.006650 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:54:08.006616 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:54:15.996580 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:54:15.996544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:54:19.012344 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:54:19.012314 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:54:19.023386 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:54:19.023360 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:54:30.392014 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:54:30.391980 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:54:41.297903 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:54:41.297826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:54:56.132795 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:54:56.132756 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:55:10.780671 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:10.780640 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:55:20.284762 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:20.284722 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:55:32.448893 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.448841 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds"] Jul 15 13:55:32.449533 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.449513 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49319879-5801-4b14-8b4d-a1bcf920c09b" containerName="authorino" Jul 15 13:55:32.449588 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.449537 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49319879-5801-4b14-8b4d-a1bcf920c09b" containerName="authorino" Jul 15 13:55:32.449655 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.449643 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49319879-5801-4b14-8b4d-a1bcf920c09b" containerName="authorino" Jul 15 13:55:32.452885 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.452866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.455093 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.455072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-044bd9bf-openshift-default-dockercfg-sbztn\"" Jul 15 13:55:32.479885 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.479863 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds"] Jul 15 13:55:32.614613 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.614577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-socket\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.614613 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.614620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-envoy\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.614876 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.614637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-credential-socket\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.614876 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.614765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-istio-token\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.614876 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.614803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r725x\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-kube-api-access-r725x\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.615029 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.614878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d25016-2c55-4357-b963-398c34431f02-istiod-ca-cert\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.615029 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.614949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-certs\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.615029 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.615018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d25016-2c55-4357-b963-398c34431f02-istio-podinfo\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.615171 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.615090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-data\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716654 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-data\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716654 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-socket\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716654 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-envoy\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716937 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-credential-socket\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716937 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-istio-token\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716937 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r725x\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-kube-api-access-r725x\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716937 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d25016-2c55-4357-b963-398c34431f02-istiod-ca-cert\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.716937 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.716931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-certs\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.717190 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.717011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d25016-2c55-4357-b963-398c34431f02-istio-podinfo\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.717190 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.717032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-socket\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.717190 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.717135 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-credential-socket\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.717354 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.717319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-certs\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.717499 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.717479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-data\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.717717 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.717693 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d25016-2c55-4357-b963-398c34431f02-istiod-ca-cert\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.718951 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.718926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-envoy\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.719327 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.719309 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d25016-2c55-4357-b963-398c34431f02-istio-podinfo\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.725073 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.725051 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-istio-token\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.725270 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.725253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r725x\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-kube-api-access-r725x\") pod \"e2e-ait-044bd9bf-openshift-default-76856459c7-brtds\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.766997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.766972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:32.905729 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.905706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds"] Jul 15 13:55:32.908010 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:55:32.907977 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d25016_2c55_4357_b963_398c34431f02.slice/crio-0b280fb62342037f5614f16d47fb512a6ac9642fcd670cabc0f88da0ff18a83b WatchSource:0}: Error finding container 0b280fb62342037f5614f16d47fb512a6ac9642fcd670cabc0f88da0ff18a83b: Status 404 returned error can't find the container with id 0b280fb62342037f5614f16d47fb512a6ac9642fcd670cabc0f88da0ff18a83b Jul 15 13:55:32.910180 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.910147 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:55:32.910247 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.910214 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:55:32.910284 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:32.910245 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:55:33.475310 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:33.475276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" event={"ID":"e1d25016-2c55-4357-b963-398c34431f02","Type":"ContainerStarted","Data":"bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c"} Jul 15 13:55:33.475748 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:33.475333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" event={"ID":"e1d25016-2c55-4357-b963-398c34431f02","Type":"ContainerStarted","Data":"0b280fb62342037f5614f16d47fb512a6ac9642fcd670cabc0f88da0ff18a83b"} Jul 15 13:55:33.496500 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:33.496450 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" podStartSLOduration=1.496431225 podStartE2EDuration="1.496431225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:55:33.495150255 +0000 UTC m=+1575.012774469" watchObservedRunningTime="2026-07-15 13:55:33.496431225 +0000 UTC m=+1575.014055428" Jul 15 13:55:33.768166 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:33.768070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:33.773484 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:33.773460 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:34.480340 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:34.480311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:34.481420 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:34.481385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:43.186975 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:43.186943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:55:50.521570 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:50.521526 2576 patch_prober.go:28] interesting pod/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 13:55:50.521995 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:50.521622 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 13:55:51.567888 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:51.567856 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d25016-2c55-4357-b963-398c34431f02" containerID="bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c" exitCode=0 Jul 15 13:55:51.568494 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:51.567933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" event={"ID":"e1d25016-2c55-4357-b963-398c34431f02","Type":"ContainerDied","Data":"bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c"} Jul 15 13:55:51.568494 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:51.568463 2576 scope.go:117] "RemoveContainer" containerID="bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c" Jul 15 13:55:51.569487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:51.569458 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:55:51.569558 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:51.569527 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:55:51.569593 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:51.569560 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:55:52.575516 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:52.575479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" event={"ID":"e1d25016-2c55-4357-b963-398c34431f02","Type":"ContainerStarted","Data":"2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0"} Jul 15 13:55:52.767891 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:52.767853 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:52.768109 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:52.768089 2576 patch_prober.go:28] interesting pod/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 13:55:52.768185 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:52.768143 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 13:55:53.768055 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:53.768014 2576 patch_prober.go:28] interesting pod/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 13:55:53.768538 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:53.768075 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 13:55:54.767850 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:54.767814 2576 patch_prober.go:28] interesting pod/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 13:55:54.768029 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:54.767886 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 13:55:55.767372 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:55.767342 2576 patch_prober.go:28] interesting pod/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 15 13:55:55.767741 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:55.767409 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 15 13:55:56.771675 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:56.771614 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:56.772130 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:56.772107 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:55:56.772666 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:55:56.772646 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:56:51.974797 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:56:51.974759 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:57:04.039067 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:04.039026 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds"] Jul 15 13:57:04.039503 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:04.039283 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" containerID="cri-o://2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0" gracePeriod=30 Jul 15 13:57:09.282565 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.282541 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:57:09.347537 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347506 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-credential-socket\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.347713 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347564 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-envoy\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.347713 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347601 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-istio-token\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.347713 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347661 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-certs\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.347713 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347704 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r725x\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-kube-api-access-r725x\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.347922 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347733 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d25016-2c55-4357-b963-398c34431f02-istio-podinfo\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.347922 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347800 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-socket\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.347922 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347829 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:57:09.347922 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347847 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-data\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.348498 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.347919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d25016-2c55-4357-b963-398c34431f02-istiod-ca-cert\") pod \"e1d25016-2c55-4357-b963-398c34431f02\" (UID: \"e1d25016-2c55-4357-b963-398c34431f02\") " Jul 15 13:57:09.348498 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.348283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:57:09.348498 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.348326 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:57:09.349487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.348688 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.349487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.348700 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d25016-2c55-4357-b963-398c34431f02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:57:09.349487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.348715 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.349487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.348747 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.349487 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.349431 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:57:09.350342 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.350298 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:57:09.350604 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.350576 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-kube-api-access-r725x" (OuterVolumeSpecName: "kube-api-access-r725x") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "kube-api-access-r725x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:57:09.351017 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.350995 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:57:09.351618 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.351596 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1d25016-2c55-4357-b963-398c34431f02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1d25016-2c55-4357-b963-398c34431f02" (UID: "e1d25016-2c55-4357-b963-398c34431f02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:57:09.450098 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.450066 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.450098 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.450094 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.450098 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.450103 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r725x\" (UniqueName: \"kubernetes.io/projected/e1d25016-2c55-4357-b963-398c34431f02-kube-api-access-r725x\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.450314 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.450112 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d25016-2c55-4357-b963-398c34431f02-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.450314 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.450120 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d25016-2c55-4357-b963-398c34431f02-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.450314 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.450129 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d25016-2c55-4357-b963-398c34431f02-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 13:57:09.945083 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.945049 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d25016-2c55-4357-b963-398c34431f02" containerID="2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0" exitCode=0 Jul 15 13:57:09.945268 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.945089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" event={"ID":"e1d25016-2c55-4357-b963-398c34431f02","Type":"ContainerDied","Data":"2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0"} Jul 15 13:57:09.945268 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.945098 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" Jul 15 13:57:09.945268 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.945133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds" event={"ID":"e1d25016-2c55-4357-b963-398c34431f02","Type":"ContainerDied","Data":"0b280fb62342037f5614f16d47fb512a6ac9642fcd670cabc0f88da0ff18a83b"} Jul 15 13:57:09.945268 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.945164 2576 scope.go:117] "RemoveContainer" containerID="2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0" Jul 15 13:57:09.954910 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.954893 2576 scope.go:117] "RemoveContainer" containerID="bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c" Jul 15 13:57:09.963532 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.963512 2576 scope.go:117] "RemoveContainer" containerID="2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0" Jul 15 13:57:09.963812 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:57:09.963793 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0\": container with ID starting with 2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0 not found: ID does not exist" containerID="2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0" Jul 15 13:57:09.963871 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.963821 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0"} err="failed to get container status \"2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0\": rpc error: code = NotFound desc = could not find container \"2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0\": container with ID starting with 2588780dd616c6b6e5a9c7b0a6c4342076530af1aaf9bf2ba0bc72e96ee9f5f0 not found: ID does not exist" Jul 15 13:57:09.963871 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.963839 2576 scope.go:117] "RemoveContainer" containerID="bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c" Jul 15 13:57:09.964057 ip-10-0-129-47 kubenswrapper[2576]: E0715 13:57:09.964038 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c\": container with ID starting with bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c not found: ID does not exist" containerID="bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c" Jul 15 13:57:09.964113 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.964067 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c"} err="failed to get container status \"bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c\": rpc error: code = NotFound desc = could not find container \"bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c\": container with ID starting with bc0d98612e980b7d65ce06e692da155a2a077383d93ba23f852bf0658ec0fb3c not found: ID does not exist" Jul 15 13:57:09.972472 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.972450 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds"] Jul 15 13:57:09.982848 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:09.982826 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-044bd9bf-openshift-default-76856459c7-brtds"] Jul 15 13:57:11.096403 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:11.096364 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1d25016-2c55-4357-b963-398c34431f02" path="/var/lib/kubelet/pods/e1d25016-2c55-4357-b963-398c34431f02/volumes" Jul 15 13:57:14.274596 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:57:14.274561 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:58:13.794461 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:58:13.794422 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:58:33.978963 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:58:33.978929 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:59:19.042978 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:19.042953 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:59:19.054872 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:19.054846 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 13:59:33.779712 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:33.779673 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 13:59:46.697641 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.697605 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d"] Jul 15 13:59:46.698095 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.698020 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" Jul 15 13:59:46.698095 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.698032 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" Jul 15 13:59:46.698095 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.698043 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" Jul 15 13:59:46.698095 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.698048 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" Jul 15 13:59:46.698297 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.698112 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" Jul 15 13:59:46.698297 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.698121 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1d25016-2c55-4357-b963-398c34431f02" containerName="istio-proxy" Jul 15 13:59:46.701164 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.701145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.703439 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.703420 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6ff521e6-openshift-default-dockercfg-8qjnn\"" Jul 15 13:59:46.718254 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.718226 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d"] Jul 15 13:59:46.819997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.819960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-token\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.819997 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.819996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-podinfo\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.820224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.820018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-certs\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.820224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.820044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-credential-socket\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.820224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.820079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72xnf\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-kube-api-access-72xnf\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.820224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.820099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-envoy\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.820224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.820142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istiod-ca-cert\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.820224 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.820219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-socket\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.820440 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.820239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-data\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921437 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-socket\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921571 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-data\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921571 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-token\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921571 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-podinfo\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921690 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-certs\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921758 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-credential-socket\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921836 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72xnf\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-kube-api-access-72xnf\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921836 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-envoy\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921836 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-socket\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.921836 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istiod-ca-cert\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.922064 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.921838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-data\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.922131 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.922055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-certs\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.922131 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.922107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-credential-socket\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.922435 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.922387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istiod-ca-cert\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.923824 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.923804 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-envoy\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.923936 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.923917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-podinfo\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.930842 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.930819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72xnf\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-kube-api-access-72xnf\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:46.930967 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:46.930848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-token\") pod \"e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:47.011972 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.011900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:47.158308 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.158191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d"] Jul 15 13:59:47.160992 ip-10-0-129-47 kubenswrapper[2576]: W0715 13:59:47.160961 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962fbd15_c1f5_446c_8a59_9fb6ccf7dcb5.slice/crio-2d1a2542c94169fed081d886f31145d858161bd48d121650cdcf934f30bb3394 WatchSource:0}: Error finding container 2d1a2542c94169fed081d886f31145d858161bd48d121650cdcf934f30bb3394: Status 404 returned error can't find the container with id 2d1a2542c94169fed081d886f31145d858161bd48d121650cdcf934f30bb3394 Jul 15 13:59:47.163303 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.163263 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:59:47.163385 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.163359 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:59:47.163459 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.163437 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 13:59:47.703502 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.703463 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" event={"ID":"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5","Type":"ContainerStarted","Data":"eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452"} Jul 15 13:59:47.703951 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.703529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" event={"ID":"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5","Type":"ContainerStarted","Data":"2d1a2542c94169fed081d886f31145d858161bd48d121650cdcf934f30bb3394"} Jul 15 13:59:47.727723 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:47.727671 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podStartSLOduration=1.7276511970000001 podStartE2EDuration="1.727651197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:59:47.72634919 +0000 UTC m=+1829.243973389" watchObservedRunningTime="2026-07-15 13:59:47.727651197 +0000 UTC m=+1829.245275396" Jul 15 13:59:48.012228 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:48.012139 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:48.016684 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:48.016661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:48.708771 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:48.708742 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:48.709672 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:48.709642 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 13:59:56.490305 ip-10-0-129-47 kubenswrapper[2576]: I0715 13:59:56.490273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:00:04.027744 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.027702 2576 patch_prober.go:28] interesting pod/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": read tcp 10.134.0.2:56842->10.134.0.34:15021: read: connection reset by peer" start-of-body= Jul 15 14:00:04.028427 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.027781 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": read tcp 10.134.0.2:56842->10.134.0.34:15021: read: connection reset by peer" Jul 15 14:00:04.788589 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.788551 2576 generic.go:356] "Generic (PLEG): container finished" podID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerID="eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452" exitCode=0 Jul 15 14:00:04.788769 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.788625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" event={"ID":"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5","Type":"ContainerDied","Data":"eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452"} Jul 15 14:00:04.789107 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.789090 2576 scope.go:117] "RemoveContainer" containerID="eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452" Jul 15 14:00:04.790200 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.790170 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:00:04.790277 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.790259 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:00:04.790315 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:04.790305 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:00:05.795044 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:05.795014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" event={"ID":"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5","Type":"ContainerStarted","Data":"5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b"} Jul 15 14:00:06.012037 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:06.012002 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 14:00:06.012341 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:06.012317 2576 patch_prober.go:28] interesting pod/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 14:00:06.012430 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:06.012380 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 14:00:07.012667 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:07.012629 2576 patch_prober.go:28] interesting pod/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 14:00:07.013023 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:07.012701 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 14:00:08.013246 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:08.013213 2576 patch_prober.go:28] interesting pod/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 14:00:08.013673 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:08.013267 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 14:00:09.012885 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:09.012850 2576 patch_prober.go:28] interesting pod/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 14:00:09.013065 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:09.012906 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 14:00:10.012682 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:10.012643 2576 patch_prober.go:28] interesting pod/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 15 14:00:10.013070 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:10.012702 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 15 14:00:11.016720 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:11.016688 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 14:00:11.017166 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:11.016963 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 14:00:11.017818 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:00:11.017798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 14:01:05.970985 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:05.970946 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:01:17.717246 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:17.717205 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d"] Jul 15 14:01:17.717839 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:17.717480 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" containerID="cri-o://5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b" gracePeriod=30 Jul 15 14:01:22.963211 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:22.963187 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 14:01:23.058727 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-podinfo\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.058862 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058756 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72xnf\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-kube-api-access-72xnf\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.058862 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-data\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.058862 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058819 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-token\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.058862 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058856 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istiod-ca-cert\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.059053 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058892 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-socket\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.059053 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058909 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-certs\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.059053 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058958 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-envoy\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.059053 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.058999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-credential-socket\") pod \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\" (UID: \"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5\") " Jul 15 14:01:23.059280 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.059201 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:01:23.059340 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.059270 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:01:23.059412 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.059376 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.059412 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.059393 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.059522 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.059410 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:01:23.059595 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.059572 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:01:23.059719 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.059663 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-data" (OuterVolumeSpecName: "istio-data") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:01:23.061013 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.060985 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-kube-api-access-72xnf" (OuterVolumeSpecName: "kube-api-access-72xnf") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "kube-api-access-72xnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:01:23.061258 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.061233 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:01:23.061339 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.061318 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:01:23.061339 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.061319 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" (UID: "962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 14:01:23.160583 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.160546 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.160583 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.160582 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.160583 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.160591 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.160830 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.160604 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.160830 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.160618 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72xnf\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-kube-api-access-72xnf\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.160830 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.160627 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.160830 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.160636 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:01:23.165947 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.165912 2576 generic.go:356] "Generic (PLEG): container finished" podID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerID="5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b" exitCode=0 Jul 15 14:01:23.166057 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.165974 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" Jul 15 14:01:23.166057 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.165997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" event={"ID":"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5","Type":"ContainerDied","Data":"5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b"} Jul 15 14:01:23.166132 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.166059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d" event={"ID":"962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5","Type":"ContainerDied","Data":"2d1a2542c94169fed081d886f31145d858161bd48d121650cdcf934f30bb3394"} Jul 15 14:01:23.166132 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.166085 2576 scope.go:117] "RemoveContainer" containerID="5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b" Jul 15 14:01:23.174775 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.174755 2576 scope.go:117] "RemoveContainer" containerID="eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452" Jul 15 14:01:23.182283 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.182265 2576 scope.go:117] "RemoveContainer" containerID="5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b" Jul 15 14:01:23.182575 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:01:23.182554 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b\": container with ID starting with 5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b not found: ID does not exist" containerID="5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b" Jul 15 14:01:23.182623 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.182583 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b"} err="failed to get container status \"5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b\": rpc error: code = NotFound desc = could not find container \"5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b\": container with ID starting with 5c7d4a768c209070173dc26c1c69632f07a2fed42914dc0f87954a906039044b not found: ID does not exist" Jul 15 14:01:23.182623 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.182602 2576 scope.go:117] "RemoveContainer" containerID="eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452" Jul 15 14:01:23.182854 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:01:23.182838 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452\": container with ID starting with eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452 not found: ID does not exist" containerID="eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452" Jul 15 14:01:23.182899 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.182860 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452"} err="failed to get container status \"eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452\": rpc error: code = NotFound desc = could not find container \"eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452\": container with ID starting with eaa6b98155acb33d318918e3a0b834fc06f8d3219b4348e3faf53603b88cb452 not found: ID does not exist" Jul 15 14:01:23.185609 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.185589 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d"] Jul 15 14:01:23.191161 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:23.191141 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6ff521e6-openshift-default-d4cf99c99-hfr2d"] Jul 15 14:01:25.096525 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:01:25.096493 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" path="/var/lib/kubelet/pods/962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5/volumes" Jul 15 14:02:43.188132 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:02:43.188094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:02:50.699458 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:02:50.699415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:02:58.190000 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:02:58.189964 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:03:08.183708 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:03:08.183659 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:03:27.378045 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:03:27.378012 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:04:00.574722 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.574641 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs"] Jul 15 14:04:00.575181 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.575146 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" Jul 15 14:04:00.575181 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.575161 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" Jul 15 14:04:00.575181 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.575170 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" Jul 15 14:04:00.575181 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.575175 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" Jul 15 14:04:00.575310 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.575253 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" Jul 15 14:04:00.575310 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.575263 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="962fbd15-c1f5-446c-8a59-9fb6ccf7dcb5" containerName="istio-proxy" Jul 15 14:04:00.578414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.578380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.580898 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.580876 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b404f66f-openshift-default-dockercfg-smbrm\"" Jul 15 14:04:00.592697 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.592674 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs"] Jul 15 14:04:00.763255 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdnv\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-kube-api-access-8kdnv\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763255 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-socket\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763486 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-istio-token\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763486 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-credential-socket\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763486 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91ed97-a051-4165-83c8-82993bd76404-istio-podinfo\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763486 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91ed97-a051-4165-83c8-82993bd76404-istiod-ca-cert\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763614 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-data\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763614 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-envoy\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.763614 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.763575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-certs\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.864790 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.864764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-istio-token\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.864949 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.864800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-credential-socket\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.864949 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.864822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91ed97-a051-4165-83c8-82993bd76404-istio-podinfo\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.864949 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.864865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91ed97-a051-4165-83c8-82993bd76404-istiod-ca-cert\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.864949 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.864900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-data\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.864949 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.864932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-envoy\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865205 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-certs\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865205 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdnv\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-kube-api-access-8kdnv\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865205 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-socket\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865357 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-credential-socket\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865434 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-certs\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865539 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865517 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-socket\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865584 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865552 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-data\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.865647 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.865632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91ed97-a051-4165-83c8-82993bd76404-istiod-ca-cert\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.867166 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.867148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-envoy\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.867505 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.867487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91ed97-a051-4165-83c8-82993bd76404-istio-podinfo\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.873884 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.873861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-istio-token\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.874025 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.873983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdnv\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-kube-api-access-8kdnv\") pod \"e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:00.888670 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:00.888645 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:01.027806 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:01.027774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs"] Jul 15 14:04:01.029802 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:04:01.029774 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e91ed97_a051_4165_83c8_82993bd76404.slice/crio-85da56cc2ef120c12e27944dd3d1b28d0c6f507d023e7bf26bf3fd866778a990 WatchSource:0}: Error finding container 85da56cc2ef120c12e27944dd3d1b28d0c6f507d023e7bf26bf3fd866778a990: Status 404 returned error can't find the container with id 85da56cc2ef120c12e27944dd3d1b28d0c6f507d023e7bf26bf3fd866778a990 Jul 15 14:04:01.032172 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:01.032140 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:04:01.032248 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:01.032208 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:04:01.032248 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:01.032241 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:04:01.921607 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:01.921571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" event={"ID":"0e91ed97-a051-4165-83c8-82993bd76404","Type":"ContainerStarted","Data":"6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e"} Jul 15 14:04:01.922005 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:01.921624 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" event={"ID":"0e91ed97-a051-4165-83c8-82993bd76404","Type":"ContainerStarted","Data":"85da56cc2ef120c12e27944dd3d1b28d0c6f507d023e7bf26bf3fd866778a990"} Jul 15 14:04:01.944143 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:01.944087 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" podStartSLOduration=1.944066888 podStartE2EDuration="1.944066888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:04:01.943577438 +0000 UTC m=+2083.461201702" watchObservedRunningTime="2026-07-15 14:04:01.944066888 +0000 UTC m=+2083.461691089" Jul 15 14:04:02.889282 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:02.889243 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:02.894088 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:02.894063 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:02.926656 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:02.926625 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:02.927613 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:02.927596 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:10.772213 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:10.772184 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:04:11.830793 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.830759 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj"] Jul 15 14:04:11.834412 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.834385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.838586 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.838568 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b404f66f\"/\"default-dockercfg-4fhj5\"" Jul 15 14:04:11.839381 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.839366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b404f66f\"/\"e2e-lifecycle-model-b404f66f-kserve-self-signed-certs\"" Jul 15 14:04:11.844481 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.844462 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b404f66f\"/\"openshift-service-ca.crt\"" Jul 15 14:04:11.846868 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.846855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b404f66f\"/\"kube-root-ca.crt\"" Jul 15 14:04:11.876691 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.876630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9vb\" (UniqueName: \"kubernetes.io/projected/66dc6585-9cf8-45de-b4aa-02a176f372b4-kube-api-access-ch9vb\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.876886 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.876736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-dshm\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.876886 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.876825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dc6585-9cf8-45de-b4aa-02a176f372b4-tls-certs\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.876886 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.876872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-tmp-dir\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.877065 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.876901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-home\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.877065 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.876940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-model-cache\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.920567 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.920528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj"] Jul 15 14:04:11.978638 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.978600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-model-cache\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.978820 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.978707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9vb\" (UniqueName: \"kubernetes.io/projected/66dc6585-9cf8-45de-b4aa-02a176f372b4-kube-api-access-ch9vb\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.978820 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.978754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-dshm\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.978940 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.978839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dc6585-9cf8-45de-b4aa-02a176f372b4-tls-certs\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.978940 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.978872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-tmp-dir\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.978940 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.978902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-home\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.979095 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.978977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-model-cache\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.979204 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.979184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-tmp-dir\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.979279 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.979261 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-home\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.981016 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.980985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-dshm\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.981330 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.981312 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dc6585-9cf8-45de-b4aa-02a176f372b4-tls-certs\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:11.994992 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:11.994972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9vb\" (UniqueName: \"kubernetes.io/projected/66dc6585-9cf8-45de-b4aa-02a176f372b4-kube-api-access-ch9vb\") pod \"e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:12.144456 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:12.144386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:12.306670 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:12.306637 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj"] Jul 15 14:04:12.308207 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:04:12.308180 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66dc6585_9cf8_45de_b4aa_02a176f372b4.slice/crio-5618738ad865faca3e12e25386ed8ed4ff4b13a46b9ebdfda75eab11373de675 WatchSource:0}: Error finding container 5618738ad865faca3e12e25386ed8ed4ff4b13a46b9ebdfda75eab11373de675: Status 404 returned error can't find the container with id 5618738ad865faca3e12e25386ed8ed4ff4b13a46b9ebdfda75eab11373de675 Jul 15 14:04:12.978460 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:12.978414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" event={"ID":"66dc6585-9cf8-45de-b4aa-02a176f372b4","Type":"ContainerStarted","Data":"3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa"} Jul 15 14:04:12.978460 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:12.978455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" event={"ID":"66dc6585-9cf8-45de-b4aa-02a176f372b4","Type":"ContainerStarted","Data":"5618738ad865faca3e12e25386ed8ed4ff4b13a46b9ebdfda75eab11373de675"} Jul 15 14:04:12.999784 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:12.999737 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" podStartSLOduration=1.999721215 podStartE2EDuration="1.999721215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:04:12.997526441 +0000 UTC m=+2094.515150652" watchObservedRunningTime="2026-07-15 14:04:12.999721215 +0000 UTC m=+2094.517345416" Jul 15 14:04:18.892280 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:18.890893 2576 patch_prober.go:28] interesting pod/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": read tcp 10.134.0.2:57800->10.134.0.35:15021: read: connection reset by peer" start-of-body= Jul 15 14:04:18.892280 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:18.890977 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": read tcp 10.134.0.2:57800->10.134.0.35:15021: read: connection reset by peer" Jul 15 14:04:19.011376 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.011339 2576 generic.go:356] "Generic (PLEG): container finished" podID="0e91ed97-a051-4165-83c8-82993bd76404" containerID="6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e" exitCode=0 Jul 15 14:04:19.011535 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.011431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" event={"ID":"0e91ed97-a051-4165-83c8-82993bd76404","Type":"ContainerDied","Data":"6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e"} Jul 15 14:04:19.011847 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.011827 2576 scope.go:117] "RemoveContainer" containerID="6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e" Jul 15 14:04:19.012963 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.012928 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:04:19.013036 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.013017 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:04:19.013076 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.013063 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:04:19.079311 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.079291 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 14:04:19.092773 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:19.092752 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 14:04:20.018331 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:20.018294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" event={"ID":"0e91ed97-a051-4165-83c8-82993bd76404","Type":"ContainerStarted","Data":"9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9"} Jul 15 14:04:20.277540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:20.277442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:04:20.889365 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:20.889333 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:20.889596 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:20.889575 2576 patch_prober.go:28] interesting pod/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 14:04:20.889645 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:20.889620 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 14:04:21.889927 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:21.889882 2576 patch_prober.go:28] interesting pod/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 14:04:21.890439 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:21.889953 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 14:04:22.144579 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:22.144471 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:22.144826 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:22.144796 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:22.157136 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:22.157110 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:22.889577 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:22.889536 2576 patch_prober.go:28] interesting pod/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 14:04:22.889787 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:22.889649 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 14:04:23.055685 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:23.055653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:04:23.889177 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:23.889143 2576 patch_prober.go:28] interesting pod/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 15 14:04:23.889356 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:23.889194 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 15 14:04:24.893144 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:24.893110 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:24.893641 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:24.893521 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:24.894119 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:24.894097 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:04:30.268682 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:30.268646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:04:50.780985 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:50.780954 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:04:56.742107 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:04:56.742069 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj"] Jul 15 14:05:11.763749 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:11.763640 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" podUID="66dc6585-9cf8-45de-b4aa-02a176f372b4" containerName="main" containerID="cri-o://3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa" gracePeriod=45 Jul 15 14:05:12.019132 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.019070 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:05:12.179098 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-dshm\") pod \"66dc6585-9cf8-45de-b4aa-02a176f372b4\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " Jul 15 14:05:12.179297 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179140 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch9vb\" (UniqueName: \"kubernetes.io/projected/66dc6585-9cf8-45de-b4aa-02a176f372b4-kube-api-access-ch9vb\") pod \"66dc6585-9cf8-45de-b4aa-02a176f372b4\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " Jul 15 14:05:12.179297 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179161 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-tmp-dir\") pod \"66dc6585-9cf8-45de-b4aa-02a176f372b4\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " Jul 15 14:05:12.179297 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179192 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-home\") pod \"66dc6585-9cf8-45de-b4aa-02a176f372b4\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " Jul 15 14:05:12.179297 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179247 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-model-cache\") pod \"66dc6585-9cf8-45de-b4aa-02a176f372b4\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " Jul 15 14:05:12.179297 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179289 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dc6585-9cf8-45de-b4aa-02a176f372b4-tls-certs\") pod \"66dc6585-9cf8-45de-b4aa-02a176f372b4\" (UID: \"66dc6585-9cf8-45de-b4aa-02a176f372b4\") " Jul 15 14:05:12.179598 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179536 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66dc6585-9cf8-45de-b4aa-02a176f372b4" (UID: "66dc6585-9cf8-45de-b4aa-02a176f372b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:12.179598 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-home" (OuterVolumeSpecName: "home") pod "66dc6585-9cf8-45de-b4aa-02a176f372b4" (UID: "66dc6585-9cf8-45de-b4aa-02a176f372b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:12.179706 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179673 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:12.179706 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179693 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:12.179841 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.179813 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "66dc6585-9cf8-45de-b4aa-02a176f372b4" (UID: "66dc6585-9cf8-45de-b4aa-02a176f372b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:12.181600 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.181568 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-dshm" (OuterVolumeSpecName: "dshm") pod "66dc6585-9cf8-45de-b4aa-02a176f372b4" (UID: "66dc6585-9cf8-45de-b4aa-02a176f372b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:12.181785 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.181758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66dc6585-9cf8-45de-b4aa-02a176f372b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66dc6585-9cf8-45de-b4aa-02a176f372b4" (UID: "66dc6585-9cf8-45de-b4aa-02a176f372b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:05:12.181842 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.181786 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66dc6585-9cf8-45de-b4aa-02a176f372b4-kube-api-access-ch9vb" (OuterVolumeSpecName: "kube-api-access-ch9vb") pod "66dc6585-9cf8-45de-b4aa-02a176f372b4" (UID: "66dc6585-9cf8-45de-b4aa-02a176f372b4"). InnerVolumeSpecName "kube-api-access-ch9vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:05:12.278867 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.278789 2576 generic.go:356] "Generic (PLEG): container finished" podID="66dc6585-9cf8-45de-b4aa-02a176f372b4" containerID="3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa" exitCode=0 Jul 15 14:05:12.278867 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.278846 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" Jul 15 14:05:12.279041 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.278881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" event={"ID":"66dc6585-9cf8-45de-b4aa-02a176f372b4","Type":"ContainerDied","Data":"3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa"} Jul 15 14:05:12.279041 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.278932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj" event={"ID":"66dc6585-9cf8-45de-b4aa-02a176f372b4","Type":"ContainerDied","Data":"5618738ad865faca3e12e25386ed8ed4ff4b13a46b9ebdfda75eab11373de675"} Jul 15 14:05:12.279041 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.278951 2576 scope.go:117] "RemoveContainer" containerID="3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa" Jul 15 14:05:12.280269 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.280246 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:12.280375 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.280271 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch9vb\" (UniqueName: \"kubernetes.io/projected/66dc6585-9cf8-45de-b4aa-02a176f372b4-kube-api-access-ch9vb\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:12.280375 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.280286 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66dc6585-9cf8-45de-b4aa-02a176f372b4-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:12.280375 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.280299 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66dc6585-9cf8-45de-b4aa-02a176f372b4-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:12.287928 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.287909 2576 scope.go:117] "RemoveContainer" containerID="3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa" Jul 15 14:05:12.288193 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:05:12.288170 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa\": container with ID starting with 3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa not found: ID does not exist" containerID="3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa" Jul 15 14:05:12.288280 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.288200 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa"} err="failed to get container status \"3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa\": rpc error: code = NotFound desc = could not find container \"3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa\": container with ID starting with 3930639d194737e0771200706a9a5ecfdafc912d587298167ec69203e33b8aaa not found: ID does not exist" Jul 15 14:05:12.301014 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.300988 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj"] Jul 15 14:05:12.305518 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:12.305498 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b404f66f/e2e-lifecycle-model-b404f66f-kserve-7f4f76ddc5-v9dvj"] Jul 15 14:05:13.096095 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:13.096060 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66dc6585-9cf8-45de-b4aa-02a176f372b4" path="/var/lib/kubelet/pods/66dc6585-9cf8-45de-b4aa-02a176f372b4/volumes" Jul 15 14:05:34.050881 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:34.050847 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs"] Jul 15 14:05:34.051490 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:34.051051 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" containerID="cri-o://9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9" gracePeriod=30 Jul 15 14:05:37.982419 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:37.982366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:05:39.303981 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.303954 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:05:39.411418 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.411370 2576 generic.go:356] "Generic (PLEG): container finished" podID="0e91ed97-a051-4165-83c8-82993bd76404" containerID="9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9" exitCode=0 Jul 15 14:05:39.411574 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.411452 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" Jul 15 14:05:39.411574 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.411458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" event={"ID":"0e91ed97-a051-4165-83c8-82993bd76404","Type":"ContainerDied","Data":"9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9"} Jul 15 14:05:39.411574 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.411513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs" event={"ID":"0e91ed97-a051-4165-83c8-82993bd76404","Type":"ContainerDied","Data":"85da56cc2ef120c12e27944dd3d1b28d0c6f507d023e7bf26bf3fd866778a990"} Jul 15 14:05:39.411574 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.411540 2576 scope.go:117] "RemoveContainer" containerID="9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9" Jul 15 14:05:39.420177 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.420164 2576 scope.go:117] "RemoveContainer" containerID="6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e" Jul 15 14:05:39.427900 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.427881 2576 scope.go:117] "RemoveContainer" containerID="9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9" Jul 15 14:05:39.428161 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:05:39.428129 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9\": container with ID starting with 9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9 not found: ID does not exist" containerID="9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9" Jul 15 14:05:39.428245 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.428166 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9"} err="failed to get container status \"9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9\": rpc error: code = NotFound desc = could not find container \"9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9\": container with ID starting with 9cc6783d6b40ab5b4bfd9ee7bb2229bf065458aa36522b83842e604c6e7532d9 not found: ID does not exist" Jul 15 14:05:39.428245 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.428182 2576 scope.go:117] "RemoveContainer" containerID="6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e" Jul 15 14:05:39.428438 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:05:39.428418 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e\": container with ID starting with 6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e not found: ID does not exist" containerID="6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e" Jul 15 14:05:39.428488 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.428445 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e"} err="failed to get container status \"6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e\": rpc error: code = NotFound desc = could not find container \"6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e\": container with ID starting with 6e8e5483c3a404015c5d01e8103581fefcd3ffcd6c625484967ddaec75d5b16e not found: ID does not exist" Jul 15 14:05:39.434739 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434723 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-credential-socket\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.434788 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434781 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91ed97-a051-4165-83c8-82993bd76404-istio-podinfo\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.434823 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434816 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91ed97-a051-4165-83c8-82993bd76404-istiod-ca-cert\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.434868 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434854 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kdnv\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-kube-api-access-8kdnv\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.434927 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434895 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-data\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.434975 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-socket\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.434975 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434935 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:39.435076 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.434988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-certs\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.435076 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435030 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-envoy\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.435076 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435067 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-istio-token\") pod \"0e91ed97-a051-4165-83c8-82993bd76404\" (UID: \"0e91ed97-a051-4165-83c8-82993bd76404\") " Jul 15 14:05:39.435266 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435239 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:39.435328 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435263 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e91ed97-a051-4165-83c8-82993bd76404-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:05:39.435456 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435434 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:39.435523 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435502 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e91ed97-a051-4165-83c8-82993bd76404-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.435523 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435521 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.435622 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435530 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.435916 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.435890 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:39.437152 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.437129 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:05:39.437261 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.437241 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:05:39.437332 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.437308 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e91ed97-a051-4165-83c8-82993bd76404-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 14:05:39.437371 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.437315 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-kube-api-access-8kdnv" (OuterVolumeSpecName: "kube-api-access-8kdnv") pod "0e91ed97-a051-4165-83c8-82993bd76404" (UID: "0e91ed97-a051-4165-83c8-82993bd76404"). InnerVolumeSpecName "kube-api-access-8kdnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:05:39.536141 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.536117 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.536141 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.536137 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.536141 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.536147 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e91ed97-a051-4165-83c8-82993bd76404-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.536347 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.536155 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.536347 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.536163 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e91ed97-a051-4165-83c8-82993bd76404-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.536347 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.536170 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kdnv\" (UniqueName: \"kubernetes.io/projected/0e91ed97-a051-4165-83c8-82993bd76404-kube-api-access-8kdnv\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:05:39.736238 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.736210 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs"] Jul 15 14:05:39.740858 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:39.740834 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b404f66f-openshift-default-58c6d57cb7-hmlhs"] Jul 15 14:05:41.096033 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:41.095997 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e91ed97-a051-4165-83c8-82993bd76404" path="/var/lib/kubelet/pods/0e91ed97-a051-4165-83c8-82993bd76404/volumes" Jul 15 14:05:52.528773 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:05:52.528729 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:06:29.767323 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767288 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9"] Jul 15 14:06:29.767847 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767742 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66dc6585-9cf8-45de-b4aa-02a176f372b4" containerName="main" Jul 15 14:06:29.767847 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767757 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="66dc6585-9cf8-45de-b4aa-02a176f372b4" containerName="main" Jul 15 14:06:29.767847 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767773 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" Jul 15 14:06:29.767847 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767779 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" Jul 15 14:06:29.767983 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767853 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" Jul 15 14:06:29.767983 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767862 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="66dc6585-9cf8-45de-b4aa-02a176f372b4" containerName="main" Jul 15 14:06:29.767983 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.767868 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" Jul 15 14:06:29.770915 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.770899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.773010 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.772988 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-16550d-openshift-default-dockercfg-8czfp\"" Jul 15 14:06:29.783905 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.783882 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9"] Jul 15 14:06:29.899848 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.899814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-socket\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900019 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.899858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istiod-ca-cert\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900019 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.899879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-podinfo\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900019 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.899978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-credential-socket\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.900037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-envoy\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.900068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-certs\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.900082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-data\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.900113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-token\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:29.900281 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:29.900137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwzw\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-kube-api-access-5dwzw\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.000999 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.000962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-certs\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.000999 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-data\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001262 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-token\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001262 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwzw\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-kube-api-access-5dwzw\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001356 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-socket\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001432 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istiod-ca-cert\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001432 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-certs\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001514 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-data\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001514 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-podinfo\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001612 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-socket\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001674 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-credential-socket\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001736 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-envoy\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.001860 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-credential-socket\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.002040 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.001985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istiod-ca-cert\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.003788 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.003770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-envoy\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.003983 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.003966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-podinfo\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.008938 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.008912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-token\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.009029 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.008945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwzw\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-kube-api-access-5dwzw\") pod \"e2e-inf-a-16550d-openshift-default-95d96796c-thxv9\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.081034 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.080962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:30.226563 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.226536 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9"] Jul 15 14:06:30.229134 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:06:30.229102 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fbe4e3_3562_48ce_ac8f_286fe87b0c3d.slice/crio-daeb74cd986718b2b8971b0b43672bb3779b69d1063c67a27902112074ed455d WatchSource:0}: Error finding container daeb74cd986718b2b8971b0b43672bb3779b69d1063c67a27902112074ed455d: Status 404 returned error can't find the container with id daeb74cd986718b2b8971b0b43672bb3779b69d1063c67a27902112074ed455d Jul 15 14:06:30.231356 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.231322 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:06:30.231435 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.231388 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:06:30.231474 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.231464 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:06:30.656073 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.656036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" event={"ID":"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d","Type":"ContainerStarted","Data":"716afd04872b3563fade31f55e88e02ef1130bff235af023fc2248503c487a0f"} Jul 15 14:06:30.656295 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.656098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" event={"ID":"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d","Type":"ContainerStarted","Data":"daeb74cd986718b2b8971b0b43672bb3779b69d1063c67a27902112074ed455d"} Jul 15 14:06:30.678791 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:30.678739 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podStartSLOduration=1.678722374 podStartE2EDuration="1.678722374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:06:30.676232206 +0000 UTC m=+2232.193856417" watchObservedRunningTime="2026-07-15 14:06:30.678722374 +0000 UTC m=+2232.196346574" Jul 15 14:06:31.081902 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:31.081810 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:31.083146 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:31.083124 2576 patch_prober.go:28] interesting pod/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 14:06:31.083240 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:31.083180 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 14:06:32.085613 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:32.085583 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:32.666108 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:32.666072 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:32.667032 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:32.667005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:40.482194 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:40.482160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:06:48.674369 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.674191 2576 patch_prober.go:28] interesting pod/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 14:06:48.674369 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.674280 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 14:06:48.753659 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.753625 2576 generic.go:356] "Generic (PLEG): container finished" podID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerID="716afd04872b3563fade31f55e88e02ef1130bff235af023fc2248503c487a0f" exitCode=0 Jul 15 14:06:48.753795 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.753672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" event={"ID":"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d","Type":"ContainerDied","Data":"716afd04872b3563fade31f55e88e02ef1130bff235af023fc2248503c487a0f"} Jul 15 14:06:48.754089 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.754075 2576 scope.go:117] "RemoveContainer" containerID="716afd04872b3563fade31f55e88e02ef1130bff235af023fc2248503c487a0f" Jul 15 14:06:48.755139 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.755109 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:06:48.755215 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.755197 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:06:48.755259 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:48.755240 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:06:49.760565 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:49.760526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" event={"ID":"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d","Type":"ContainerStarted","Data":"3d6769b9a1441c63f0aecdc545556e4f5744ce1300f2fbb53a6ff6d2e578ca2a"} Jul 15 14:06:50.082152 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:50.082055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:50.082302 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:50.082269 2576 patch_prober.go:28] interesting pod/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 14:06:50.082344 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:50.082314 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 14:06:51.081693 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:51.081660 2576 patch_prober.go:28] interesting pod/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 14:06:51.082082 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:51.081726 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 14:06:52.081765 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:52.081724 2576 patch_prober.go:28] interesting pod/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 14:06:52.082227 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:52.081786 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 14:06:53.081903 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:53.081866 2576 patch_prober.go:28] interesting pod/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 14:06:53.082372 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:53.081931 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 14:06:53.374103 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:53.374066 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:06:54.085506 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:54.085475 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:54.085946 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:54.085822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:06:54.086660 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:06:54.086641 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:07:01.188727 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.188694 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p"] Jul 15 14:07:01.189110 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.189102 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" Jul 15 14:07:01.189158 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.189114 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e91ed97-a051-4165-83c8-82993bd76404" containerName="istio-proxy" Jul 15 14:07:01.191215 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.191194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.193994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.193973 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16550d\"/\"kube-root-ca.crt\"" Jul 15 14:07:01.195691 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.195671 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16550d\"/\"openshift-service-ca.crt\"" Jul 15 14:07:01.196120 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.196102 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16550d\"/\"default-dockercfg-fljmf\"" Jul 15 14:07:01.204090 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.204072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16550d\"/\"test-model-16550d-kserve-self-signed-certs\"" Jul 15 14:07:01.212733 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.212710 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p"] Jul 15 14:07:01.308981 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.308935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-tmp-dir\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.309157 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.309032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-home\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.309157 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.309072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-dshm\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.309157 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.309122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-model-cache\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.309325 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.309166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff95c57-d01b-4416-99e6-16cb827fb55d-tls-certs\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.309325 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.309264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpw6z\" (UniqueName: \"kubernetes.io/projected/9ff95c57-d01b-4416-99e6-16cb827fb55d-kube-api-access-mpw6z\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410028 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.409993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-tmp-dir\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410217 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-home\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410217 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-dshm\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410350 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-model-cache\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410350 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff95c57-d01b-4416-99e6-16cb827fb55d-tls-certs\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410350 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpw6z\" (UniqueName: \"kubernetes.io/projected/9ff95c57-d01b-4416-99e6-16cb827fb55d-kube-api-access-mpw6z\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410518 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-home\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410518 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410380 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-tmp-dir\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.410628 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.410611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-model-cache\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.412497 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.412477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-dshm\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.412721 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.412704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff95c57-d01b-4416-99e6-16cb827fb55d-tls-certs\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.418586 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.418565 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpw6z\" (UniqueName: \"kubernetes.io/projected/9ff95c57-d01b-4416-99e6-16cb827fb55d-kube-api-access-mpw6z\") pod \"test-model-16550d-kserve-5bb8b6579f-7vj8p\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.501255 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.501174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:01.639662 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.639636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p"] Jul 15 14:07:01.641528 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:07:01.641498 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ff95c57_d01b_4416_99e6_16cb827fb55d.slice/crio-47c1020f7ba1672c5d8bbb34b7b64cc47d64e48a489e250e84f925520ef7e3f1 WatchSource:0}: Error finding container 47c1020f7ba1672c5d8bbb34b7b64cc47d64e48a489e250e84f925520ef7e3f1: Status 404 returned error can't find the container with id 47c1020f7ba1672c5d8bbb34b7b64cc47d64e48a489e250e84f925520ef7e3f1 Jul 15 14:07:01.823195 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.823096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" event={"ID":"9ff95c57-d01b-4416-99e6-16cb827fb55d","Type":"ContainerStarted","Data":"95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b"} Jul 15 14:07:01.823195 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.823141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" event={"ID":"9ff95c57-d01b-4416-99e6-16cb827fb55d","Type":"ContainerStarted","Data":"47c1020f7ba1672c5d8bbb34b7b64cc47d64e48a489e250e84f925520ef7e3f1"} Jul 15 14:07:01.842156 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:01.842100 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" podStartSLOduration=0.842082412 podStartE2EDuration="842.082412ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:07:01.840374181 +0000 UTC m=+2263.357998382" watchObservedRunningTime="2026-07-15 14:07:01.842082412 +0000 UTC m=+2263.359706611" Jul 15 14:07:06.376226 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:06.376194 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:07:11.502105 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:11.502068 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:11.502627 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:11.502145 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:11.514802 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:11.514777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:11.889850 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:11.889822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:16.973294 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:16.973259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:07:22.881291 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.881257 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2"] Jul 15 14:07:22.883735 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.883716 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:22.886346 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.886323 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-59923e\"/\"kube-root-ca.crt\"" Jul 15 14:07:22.886461 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.886342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-59923e\"/\"default-dockercfg-xldnm\"" Jul 15 14:07:22.886461 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.886342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-59923e\"/\"test-model-59923e-kserve-self-signed-certs\"" Jul 15 14:07:22.886461 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.886327 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-59923e\"/\"openshift-service-ca.crt\"" Jul 15 14:07:22.893976 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.893952 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2"] Jul 15 14:07:22.927199 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.927164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-home\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:22.927316 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.927222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-dshm\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:22.927316 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.927253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-tmp-dir\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:22.927421 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.927322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-model-cache\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:22.927421 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.927355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwm7\" (UniqueName: \"kubernetes.io/projected/e26658e6-caac-4c85-adfa-7e99b897b938-kube-api-access-9rwm7\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:22.927421 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:22.927377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26658e6-caac-4c85-adfa-7e99b897b938-tls-certs\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.028754 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.028713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-model-cache\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.028946 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.028764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwm7\" (UniqueName: \"kubernetes.io/projected/e26658e6-caac-4c85-adfa-7e99b897b938-kube-api-access-9rwm7\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.028946 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.028795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26658e6-caac-4c85-adfa-7e99b897b938-tls-certs\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.028946 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.028821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-home\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.028946 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.028854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-dshm\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.028946 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.028871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-tmp-dir\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.029205 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.029153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-model-cache\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.029205 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.029176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-tmp-dir\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.029310 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.029254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-home\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.031056 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.031032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-dshm\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.031352 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.031333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26658e6-caac-4c85-adfa-7e99b897b938-tls-certs\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.037124 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.037100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwm7\" (UniqueName: \"kubernetes.io/projected/e26658e6-caac-4c85-adfa-7e99b897b938-kube-api-access-9rwm7\") pod \"test-model-59923e-kserve-c6b556ddc-hpmg2\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.193382 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.193299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:23.324329 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.324278 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2"] Jul 15 14:07:23.327029 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:07:23.327001 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode26658e6_caac_4c85_adfa_7e99b897b938.slice/crio-1366826f99ea587c17c2cb81720df0b78f2e58f9dc23b78bfa87d97d68988c58 WatchSource:0}: Error finding container 1366826f99ea587c17c2cb81720df0b78f2e58f9dc23b78bfa87d97d68988c58: Status 404 returned error can't find the container with id 1366826f99ea587c17c2cb81720df0b78f2e58f9dc23b78bfa87d97d68988c58 Jul 15 14:07:23.939515 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.939476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" event={"ID":"e26658e6-caac-4c85-adfa-7e99b897b938","Type":"ContainerStarted","Data":"4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801"} Jul 15 14:07:23.939515 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.939521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" event={"ID":"e26658e6-caac-4c85-adfa-7e99b897b938","Type":"ContainerStarted","Data":"1366826f99ea587c17c2cb81720df0b78f2e58f9dc23b78bfa87d97d68988c58"} Jul 15 14:07:23.958280 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:23.958217 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" podStartSLOduration=1.958194633 podStartE2EDuration="1.958194633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:07:23.957566525 +0000 UTC m=+2285.475190747" watchObservedRunningTime="2026-07-15 14:07:23.958194633 +0000 UTC m=+2285.475818851" Jul 15 14:07:27.869765 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:27.869731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:07:33.194296 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:33.194263 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:33.194743 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:33.194516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:33.206787 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:33.206762 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:34.000281 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:34.000243 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:07:41.776375 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:41.776334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:07:44.117177 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:44.117142 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p"] Jul 15 14:07:45.061254 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:45.061208 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2"] Jul 15 14:07:58.978050 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:58.978013 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:07:59.137643 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.137603 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" podUID="9ff95c57-d01b-4416-99e6-16cb827fb55d" containerName="main" containerID="cri-o://95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b" gracePeriod=45 Jul 15 14:07:59.391745 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.391720 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:07:59.478607 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.478580 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-home\") pod \"9ff95c57-d01b-4416-99e6-16cb827fb55d\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " Jul 15 14:07:59.478759 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.478612 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff95c57-d01b-4416-99e6-16cb827fb55d-tls-certs\") pod \"9ff95c57-d01b-4416-99e6-16cb827fb55d\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " Jul 15 14:07:59.478759 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.478669 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-dshm\") pod \"9ff95c57-d01b-4416-99e6-16cb827fb55d\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " Jul 15 14:07:59.478759 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.478720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-tmp-dir\") pod \"9ff95c57-d01b-4416-99e6-16cb827fb55d\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " Jul 15 14:07:59.478878 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.478762 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpw6z\" (UniqueName: \"kubernetes.io/projected/9ff95c57-d01b-4416-99e6-16cb827fb55d-kube-api-access-mpw6z\") pod \"9ff95c57-d01b-4416-99e6-16cb827fb55d\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " Jul 15 14:07:59.478878 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.478797 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-model-cache\") pod \"9ff95c57-d01b-4416-99e6-16cb827fb55d\" (UID: \"9ff95c57-d01b-4416-99e6-16cb827fb55d\") " Jul 15 14:07:59.478969 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.478886 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-home" (OuterVolumeSpecName: "home") pod "9ff95c57-d01b-4416-99e6-16cb827fb55d" (UID: "9ff95c57-d01b-4416-99e6-16cb827fb55d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:07:59.479102 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.479071 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ff95c57-d01b-4416-99e6-16cb827fb55d" (UID: "9ff95c57-d01b-4416-99e6-16cb827fb55d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:07:59.479229 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.479097 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ff95c57-d01b-4416-99e6-16cb827fb55d" (UID: "9ff95c57-d01b-4416-99e6-16cb827fb55d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:07:59.479294 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.479227 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:07:59.480878 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.480859 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff95c57-d01b-4416-99e6-16cb827fb55d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ff95c57-d01b-4416-99e6-16cb827fb55d" (UID: "9ff95c57-d01b-4416-99e6-16cb827fb55d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:07:59.480950 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.480904 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff95c57-d01b-4416-99e6-16cb827fb55d-kube-api-access-mpw6z" (OuterVolumeSpecName: "kube-api-access-mpw6z") pod "9ff95c57-d01b-4416-99e6-16cb827fb55d" (UID: "9ff95c57-d01b-4416-99e6-16cb827fb55d"). InnerVolumeSpecName "kube-api-access-mpw6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:07:59.480950 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.480919 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-dshm" (OuterVolumeSpecName: "dshm") pod "9ff95c57-d01b-4416-99e6-16cb827fb55d" (UID: "9ff95c57-d01b-4416-99e6-16cb827fb55d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:07:59.580687 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.580608 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:07:59.580687 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.580634 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpw6z\" (UniqueName: \"kubernetes.io/projected/9ff95c57-d01b-4416-99e6-16cb827fb55d-kube-api-access-mpw6z\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:07:59.580687 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.580644 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:07:59.580687 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.580653 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff95c57-d01b-4416-99e6-16cb827fb55d-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:07:59.580687 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:07:59.580660 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff95c57-d01b-4416-99e6-16cb827fb55d-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:00.082046 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.081976 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" podUID="e26658e6-caac-4c85-adfa-7e99b897b938" containerName="main" containerID="cri-o://4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801" gracePeriod=45 Jul 15 14:08:00.121540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.121502 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ff95c57-d01b-4416-99e6-16cb827fb55d" containerID="95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b" exitCode=0 Jul 15 14:08:00.121540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.121548 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" event={"ID":"9ff95c57-d01b-4416-99e6-16cb827fb55d","Type":"ContainerDied","Data":"95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b"} Jul 15 14:08:00.121540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.121556 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" Jul 15 14:08:00.121923 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.121579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p" event={"ID":"9ff95c57-d01b-4416-99e6-16cb827fb55d","Type":"ContainerDied","Data":"47c1020f7ba1672c5d8bbb34b7b64cc47d64e48a489e250e84f925520ef7e3f1"} Jul 15 14:08:00.121923 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.121597 2576 scope.go:117] "RemoveContainer" containerID="95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b" Jul 15 14:08:00.151448 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.151422 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p"] Jul 15 14:08:00.154898 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.154876 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-16550d/test-model-16550d-kserve-5bb8b6579f-7vj8p"] Jul 15 14:08:00.194013 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.193750 2576 scope.go:117] "RemoveContainer" containerID="95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b" Jul 15 14:08:00.194082 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:08:00.194059 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b\": container with ID starting with 95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b not found: ID does not exist" containerID="95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b" Jul 15 14:08:00.194134 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.194089 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b"} err="failed to get container status \"95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b\": rpc error: code = NotFound desc = could not find container \"95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b\": container with ID starting with 95fcbfb30bde1c31b73debd96450140043a8999a8f41b89b87c94bfcd5c3994b not found: ID does not exist" Jul 15 14:08:00.340344 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.340286 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:08:00.386725 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.386698 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26658e6-caac-4c85-adfa-7e99b897b938-tls-certs\") pod \"e26658e6-caac-4c85-adfa-7e99b897b938\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " Jul 15 14:08:00.386873 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.386749 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-dshm\") pod \"e26658e6-caac-4c85-adfa-7e99b897b938\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " Jul 15 14:08:00.386873 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.386821 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rwm7\" (UniqueName: \"kubernetes.io/projected/e26658e6-caac-4c85-adfa-7e99b897b938-kube-api-access-9rwm7\") pod \"e26658e6-caac-4c85-adfa-7e99b897b938\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " Jul 15 14:08:00.386987 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.386908 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-home\") pod \"e26658e6-caac-4c85-adfa-7e99b897b938\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " Jul 15 14:08:00.386987 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.386955 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-tmp-dir\") pod \"e26658e6-caac-4c85-adfa-7e99b897b938\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " Jul 15 14:08:00.387091 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.386997 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-model-cache\") pod \"e26658e6-caac-4c85-adfa-7e99b897b938\" (UID: \"e26658e6-caac-4c85-adfa-7e99b897b938\") " Jul 15 14:08:00.387192 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.387166 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-home" (OuterVolumeSpecName: "home") pod "e26658e6-caac-4c85-adfa-7e99b897b938" (UID: "e26658e6-caac-4c85-adfa-7e99b897b938"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:00.387261 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.387186 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e26658e6-caac-4c85-adfa-7e99b897b938" (UID: "e26658e6-caac-4c85-adfa-7e99b897b938"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:00.387351 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.387321 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-model-cache" (OuterVolumeSpecName: "model-cache") pod "e26658e6-caac-4c85-adfa-7e99b897b938" (UID: "e26658e6-caac-4c85-adfa-7e99b897b938"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:00.387497 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.387338 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:00.387497 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.387418 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:00.389110 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.389087 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26658e6-caac-4c85-adfa-7e99b897b938-kube-api-access-9rwm7" (OuterVolumeSpecName: "kube-api-access-9rwm7") pod "e26658e6-caac-4c85-adfa-7e99b897b938" (UID: "e26658e6-caac-4c85-adfa-7e99b897b938"). InnerVolumeSpecName "kube-api-access-9rwm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:08:00.389110 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.389096 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-dshm" (OuterVolumeSpecName: "dshm") pod "e26658e6-caac-4c85-adfa-7e99b897b938" (UID: "e26658e6-caac-4c85-adfa-7e99b897b938"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:00.389110 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.389105 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26658e6-caac-4c85-adfa-7e99b897b938-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e26658e6-caac-4c85-adfa-7e99b897b938" (UID: "e26658e6-caac-4c85-adfa-7e99b897b938"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:08:00.488766 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.488730 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:00.488766 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.488760 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26658e6-caac-4c85-adfa-7e99b897b938-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:00.488766 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.488768 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26658e6-caac-4c85-adfa-7e99b897b938-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:00.488766 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:00.488776 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rwm7\" (UniqueName: \"kubernetes.io/projected/e26658e6-caac-4c85-adfa-7e99b897b938-kube-api-access-9rwm7\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:01.096058 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.096014 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ff95c57-d01b-4416-99e6-16cb827fb55d" path="/var/lib/kubelet/pods/9ff95c57-d01b-4416-99e6-16cb827fb55d/volumes" Jul 15 14:08:01.127432 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.127388 2576 generic.go:356] "Generic (PLEG): container finished" podID="e26658e6-caac-4c85-adfa-7e99b897b938" containerID="4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801" exitCode=0 Jul 15 14:08:01.127584 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.127442 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" Jul 15 14:08:01.127584 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.127477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" event={"ID":"e26658e6-caac-4c85-adfa-7e99b897b938","Type":"ContainerDied","Data":"4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801"} Jul 15 14:08:01.127584 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.127525 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2" event={"ID":"e26658e6-caac-4c85-adfa-7e99b897b938","Type":"ContainerDied","Data":"1366826f99ea587c17c2cb81720df0b78f2e58f9dc23b78bfa87d97d68988c58"} Jul 15 14:08:01.127584 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.127551 2576 scope.go:117] "RemoveContainer" containerID="4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801" Jul 15 14:08:01.136163 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.136146 2576 scope.go:117] "RemoveContainer" containerID="4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801" Jul 15 14:08:01.136435 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:08:01.136413 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801\": container with ID starting with 4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801 not found: ID does not exist" containerID="4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801" Jul 15 14:08:01.136491 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.136444 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801"} err="failed to get container status \"4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801\": rpc error: code = NotFound desc = could not find container \"4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801\": container with ID starting with 4cc94c7584a20ffa47b7c7aa6a7618498762df8b490f7953af91432bc8417801 not found: ID does not exist" Jul 15 14:08:01.143521 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.143502 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2"] Jul 15 14:08:01.149692 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:01.149673 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-59923e/test-model-59923e-kserve-c6b556ddc-hpmg2"] Jul 15 14:08:03.096379 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:03.096345 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26658e6-caac-4c85-adfa-7e99b897b938" path="/var/lib/kubelet/pods/e26658e6-caac-4c85-adfa-7e99b897b938/volumes" Jul 15 14:08:11.006317 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:11.006232 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9"] Jul 15 14:08:11.006705 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:11.006460 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" containerID="cri-o://3d6769b9a1441c63f0aecdc545556e4f5744ce1300f2fbb53a6ff6d2e578ca2a" gracePeriod=30 Jul 15 14:08:16.207168 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.207129 2576 generic.go:356] "Generic (PLEG): container finished" podID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerID="3d6769b9a1441c63f0aecdc545556e4f5744ce1300f2fbb53a6ff6d2e578ca2a" exitCode=0 Jul 15 14:08:16.207610 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.207200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" event={"ID":"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d","Type":"ContainerDied","Data":"3d6769b9a1441c63f0aecdc545556e4f5744ce1300f2fbb53a6ff6d2e578ca2a"} Jul 15 14:08:16.207610 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.207297 2576 scope.go:117] "RemoveContainer" containerID="716afd04872b3563fade31f55e88e02ef1130bff235af023fc2248503c487a0f" Jul 15 14:08:16.260806 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.260778 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:08:16.443051 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-token\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443249 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-credential-socket\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443249 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443098 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-envoy\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443249 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443210 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dwzw\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-kube-api-access-5dwzw\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443444 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443259 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-data\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443514 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443452 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istiod-ca-cert\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443514 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443476 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:16.443624 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443516 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-podinfo\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443624 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-socket\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443728 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443635 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-certs\") pod \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\" (UID: \"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d\") " Jul 15 14:08:16.443807 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443786 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:08:16.443985 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443963 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:16.444062 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.443983 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:16.444122 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.444103 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.444122 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.444116 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.444235 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.444124 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.444235 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.444137 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.444342 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.444259 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:16.445474 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.445446 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:08:16.445719 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.445696 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 14:08:16.445801 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.445708 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-kube-api-access-5dwzw" (OuterVolumeSpecName: "kube-api-access-5dwzw") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "kube-api-access-5dwzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:08:16.445849 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.445828 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" (UID: "37fbe4e3-3562-48ce-ac8f-286fe87b0c3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:08:16.545103 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.545069 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.545103 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.545097 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwzw\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-kube-api-access-5dwzw\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.545277 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.545110 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.545277 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.545124 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:16.545277 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:16.545135 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:08:17.213267 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:17.213243 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" Jul 15 14:08:17.213267 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:17.213260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9" event={"ID":"37fbe4e3-3562-48ce-ac8f-286fe87b0c3d","Type":"ContainerDied","Data":"daeb74cd986718b2b8971b0b43672bb3779b69d1063c67a27902112074ed455d"} Jul 15 14:08:17.213725 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:17.213318 2576 scope.go:117] "RemoveContainer" containerID="3d6769b9a1441c63f0aecdc545556e4f5744ce1300f2fbb53a6ff6d2e578ca2a" Jul 15 14:08:17.241050 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:17.241023 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9"] Jul 15 14:08:17.244926 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:17.244906 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-16550d-openshift-default-95d96796c-thxv9"] Jul 15 14:08:19.096722 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:19.096690 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" path="/var/lib/kubelet/pods/37fbe4e3-3562-48ce-ac8f-286fe87b0c3d/volumes" Jul 15 14:08:24.573919 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:24.573879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:08:38.183878 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.183844 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv"] Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184228 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184239 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184249 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184255 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184262 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26658e6-caac-4c85-adfa-7e99b897b938" containerName="main" Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184268 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26658e6-caac-4c85-adfa-7e99b897b938" containerName="main" Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184280 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff95c57-d01b-4416-99e6-16cb827fb55d" containerName="main" Jul 15 14:08:38.184337 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184285 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff95c57-d01b-4416-99e6-16cb827fb55d" containerName="main" Jul 15 14:08:38.184634 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184350 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" Jul 15 14:08:38.184634 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184358 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ff95c57-d01b-4416-99e6-16cb827fb55d" containerName="main" Jul 15 14:08:38.184634 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.184363 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26658e6-caac-4c85-adfa-7e99b897b938" containerName="main" Jul 15 14:08:38.187517 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.187499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.190090 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.190068 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-69a378-openshift-default-dockercfg-wcl8z\"" Jul 15 14:08:38.206623 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.206599 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv"] Jul 15 14:08:38.253853 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.253821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-certs\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.253995 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.253864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjgt\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-kube-api-access-ssjgt\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.253995 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.253919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04152116-ceb1-496d-8e01-adb39518aebf-istiod-ca-cert\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.253995 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.253946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-data\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.253995 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.253973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-credential-socket\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.254206 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.254026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-socket\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.254206 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.254119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-envoy\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.254206 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.254159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-istio-token\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.254306 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.254228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04152116-ceb1-496d-8e01-adb39518aebf-istio-podinfo\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355532 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04152116-ceb1-496d-8e01-adb39518aebf-istiod-ca-cert\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355532 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-data\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355746 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-credential-socket\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355746 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-socket\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355746 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-envoy\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355746 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-istio-token\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355746 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04152116-ceb1-496d-8e01-adb39518aebf-istio-podinfo\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355991 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-certs\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355991 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjgt\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-kube-api-access-ssjgt\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.355991 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.355975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-socket\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.356180 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.356154 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-credential-socket\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.356257 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.356199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-data\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.356325 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.356307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-certs\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.356462 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.356440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04152116-ceb1-496d-8e01-adb39518aebf-istiod-ca-cert\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.358054 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.358031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-envoy\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.358259 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.358239 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04152116-ceb1-496d-8e01-adb39518aebf-istio-podinfo\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.364664 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.364640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-istio-token\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.367607 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.367589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjgt\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-kube-api-access-ssjgt\") pod \"e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.497768 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.497672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:38.637019 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.636982 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv"] Jul 15 14:08:38.640202 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:08:38.640180 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04152116_ceb1_496d_8e01_adb39518aebf.slice/crio-d9a0514299458cf09b5744a8ccfaa222e855562a7169b476727d99dcb3bdfb4f WatchSource:0}: Error finding container d9a0514299458cf09b5744a8ccfaa222e855562a7169b476727d99dcb3bdfb4f: Status 404 returned error can't find the container with id d9a0514299458cf09b5744a8ccfaa222e855562a7169b476727d99dcb3bdfb4f Jul 15 14:08:38.642286 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.642257 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:08:38.642347 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.642319 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:08:38.642384 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:38.642348 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:08:39.319488 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:39.319446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" event={"ID":"04152116-ceb1-496d-8e01-adb39518aebf","Type":"ContainerStarted","Data":"193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4"} Jul 15 14:08:39.319995 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:39.319512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" event={"ID":"04152116-ceb1-496d-8e01-adb39518aebf","Type":"ContainerStarted","Data":"d9a0514299458cf09b5744a8ccfaa222e855562a7169b476727d99dcb3bdfb4f"} Jul 15 14:08:39.358573 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:39.358527 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" podStartSLOduration=1.358510951 podStartE2EDuration="1.358510951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:08:39.35796286 +0000 UTC m=+2360.875587072" watchObservedRunningTime="2026-07-15 14:08:39.358510951 +0000 UTC m=+2360.876135151" Jul 15 14:08:39.498550 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:39.498516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:39.503063 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:39.503039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:40.325071 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:40.325031 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:40.325981 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:40.325959 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:46.386146 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:46.386108 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:08:54.395895 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:54.395860 2576 generic.go:356] "Generic (PLEG): container finished" podID="04152116-ceb1-496d-8e01-adb39518aebf" containerID="193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4" exitCode=0 Jul 15 14:08:54.396231 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:54.395913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" event={"ID":"04152116-ceb1-496d-8e01-adb39518aebf","Type":"ContainerDied","Data":"193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4"} Jul 15 14:08:54.396345 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:54.396330 2576 scope.go:117] "RemoveContainer" containerID="193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4" Jul 15 14:08:54.397358 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:54.397330 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:08:54.397438 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:54.397426 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:08:54.397498 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:54.397484 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:08:55.325836 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:55.325794 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:55.402496 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:55.402464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" event={"ID":"04152116-ceb1-496d-8e01-adb39518aebf","Type":"ContainerStarted","Data":"504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555"} Jul 15 14:08:55.498125 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:55.498086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:55.499155 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:55.499128 2576 patch_prober.go:28] interesting pod/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 14:08:55.499277 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:55.499191 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 14:08:56.498850 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:56.498815 2576 patch_prober.go:28] interesting pod/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 14:08:56.499228 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:56.498870 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 14:08:57.498870 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:57.498833 2576 patch_prober.go:28] interesting pod/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 14:08:57.499259 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:57.498887 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 14:08:58.498591 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:58.498554 2576 patch_prober.go:28] interesting pod/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 14:08:58.498772 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:58.498608 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 14:08:58.879140 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:58.879098 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:08:59.502644 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:59.502616 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:59.503023 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:59.503002 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:08:59.503779 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:08:59.503761 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:09:16.183443 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:16.183390 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:09:19.116320 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:19.116292 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 14:09:19.126668 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:19.126644 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 14:09:28.484429 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:28.484378 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv"] Jul 15 14:09:28.484818 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:28.484624 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" containerID="cri-o://504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555" gracePeriod=30 Jul 15 14:09:29.505330 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:29.505293 2576 patch_prober.go:28] interesting pod/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 14:09:29.505711 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:29.505358 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 14:09:33.730195 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.730171 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:09:33.897319 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897279 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04152116-ceb1-496d-8e01-adb39518aebf-istio-podinfo\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897550 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897332 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-envoy\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897550 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897375 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-socket\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897550 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897442 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssjgt\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-kube-api-access-ssjgt\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897550 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897483 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-credential-socket\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897853 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897578 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04152116-ceb1-496d-8e01-adb39518aebf-istiod-ca-cert\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897853 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897619 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-data\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897853 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897642 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-istio-token\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897853 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897696 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-certs\") pod \"04152116-ceb1-496d-8e01-adb39518aebf\" (UID: \"04152116-ceb1-496d-8e01-adb39518aebf\") " Jul 15 14:09:33.897853 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897735 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:09:33.897853 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.897808 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:09:33.898147 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.898020 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:09:33.898147 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.898032 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04152116-ceb1-496d-8e01-adb39518aebf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:09:33.898147 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.898088 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.898147 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.898110 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.898468 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.898446 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-data" (OuterVolumeSpecName: "istio-data") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:09:33.899750 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.899726 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-istio-token" (OuterVolumeSpecName: "istio-token") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:09:33.899848 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.899732 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:09:33.900027 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.899992 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-kube-api-access-ssjgt" (OuterVolumeSpecName: "kube-api-access-ssjgt") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "kube-api-access-ssjgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:09:33.900115 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.900056 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04152116-ceb1-496d-8e01-adb39518aebf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04152116-ceb1-496d-8e01-adb39518aebf" (UID: "04152116-ceb1-496d-8e01-adb39518aebf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 14:09:33.999006 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.998969 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjgt\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-kube-api-access-ssjgt\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.999006 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.998998 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04152116-ceb1-496d-8e01-adb39518aebf-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.999006 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.999011 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.999246 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.999023 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04152116-ceb1-496d-8e01-adb39518aebf-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.999246 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.999033 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.999246 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.999047 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04152116-ceb1-496d-8e01-adb39518aebf-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:33.999246 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:33.999057 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04152116-ceb1-496d-8e01-adb39518aebf-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:09:34.600544 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.600506 2576 generic.go:356] "Generic (PLEG): container finished" podID="04152116-ceb1-496d-8e01-adb39518aebf" containerID="504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555" exitCode=0 Jul 15 14:09:34.600725 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.600560 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" Jul 15 14:09:34.600725 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.600591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" event={"ID":"04152116-ceb1-496d-8e01-adb39518aebf","Type":"ContainerDied","Data":"504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555"} Jul 15 14:09:34.600725 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.600651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv" event={"ID":"04152116-ceb1-496d-8e01-adb39518aebf","Type":"ContainerDied","Data":"d9a0514299458cf09b5744a8ccfaa222e855562a7169b476727d99dcb3bdfb4f"} Jul 15 14:09:34.600725 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.600673 2576 scope.go:117] "RemoveContainer" containerID="504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555" Jul 15 14:09:34.609782 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.609764 2576 scope.go:117] "RemoveContainer" containerID="193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4" Jul 15 14:09:34.617367 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.617352 2576 scope.go:117] "RemoveContainer" containerID="504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555" Jul 15 14:09:34.617669 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:09:34.617651 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555\": container with ID starting with 504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555 not found: ID does not exist" containerID="504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555" Jul 15 14:09:34.617763 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.617680 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555"} err="failed to get container status \"504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555\": rpc error: code = NotFound desc = could not find container \"504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555\": container with ID starting with 504ed37c24adf37c61fc90daafcb656345264f7eb585f9bd8c6797fffffac555 not found: ID does not exist" Jul 15 14:09:34.617763 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.617714 2576 scope.go:117] "RemoveContainer" containerID="193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4" Jul 15 14:09:34.617993 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:09:34.617976 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4\": container with ID starting with 193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4 not found: ID does not exist" containerID="193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4" Jul 15 14:09:34.618035 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.617999 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4"} err="failed to get container status \"193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4\": rpc error: code = NotFound desc = could not find container \"193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4\": container with ID starting with 193fd61d1f6f456ac13bc587f9ab7dfa3476b1710722de74cc6467783e7480f4 not found: ID does not exist" Jul 15 14:09:34.622105 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.622082 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv"] Jul 15 14:09:34.631440 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:34.627169 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-69a378-openshift-default-6b5448859f-r8bvv"] Jul 15 14:09:35.096190 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:35.096150 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04152116-ceb1-496d-8e01-adb39518aebf" path="/var/lib/kubelet/pods/04152116-ceb1-496d-8e01-adb39518aebf/volumes" Jul 15 14:09:42.582260 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:42.582180 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:09:55.777049 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777011 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd"] Jul 15 14:09:55.777658 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777472 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" Jul 15 14:09:55.777658 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777488 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" Jul 15 14:09:55.777658 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777516 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" Jul 15 14:09:55.777658 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777522 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" Jul 15 14:09:55.777658 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777592 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="37fbe4e3-3562-48ce-ac8f-286fe87b0c3d" containerName="istio-proxy" Jul 15 14:09:55.777658 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777604 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" Jul 15 14:09:55.777658 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.777610 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="04152116-ceb1-496d-8e01-adb39518aebf" containerName="istio-proxy" Jul 15 14:09:55.782277 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.782256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.784617 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.784599 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-2723e9-openshift-default-dockercfg-ss2kb\"" Jul 15 14:09:55.796454 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.796427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd"] Jul 15 14:09:55.902470 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-envoy\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.902662 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cb802c-3122-4e4d-8504-0394abd7877a-istiod-ca-cert\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.902662 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-data\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.902662 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kll7x\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-kube-api-access-kll7x\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.902774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-certs\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.902774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cb802c-3122-4e4d-8504-0394abd7877a-istio-podinfo\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.902774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-istio-token\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.902774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-credential-socket\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:55.903014 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:55.902805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-socket\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003730 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cb802c-3122-4e4d-8504-0394abd7877a-istiod-ca-cert\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003730 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-data\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kll7x\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-kube-api-access-kll7x\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-certs\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cb802c-3122-4e4d-8504-0394abd7877a-istio-podinfo\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-istio-token\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-credential-socket\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-socket\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.003988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.003920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-envoy\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.004370 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.004263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-certs\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.004471 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.004373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-credential-socket\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.004570 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.004548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-socket\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.004636 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.004571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-data\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.004636 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.004588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cb802c-3122-4e4d-8504-0394abd7877a-istiod-ca-cert\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.006153 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.006132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-envoy\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.006258 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.006217 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cb802c-3122-4e4d-8504-0394abd7877a-istio-podinfo\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.012727 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.012701 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-istio-token\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.012827 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.012780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kll7x\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-kube-api-access-kll7x\") pod \"e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.092674 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.092588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:56.229346 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.229313 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd"] Jul 15 14:09:56.232782 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:09:56.232756 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cb802c_3122_4e4d_8504_0394abd7877a.slice/crio-11fce052ec3570fb9b0c25f6f07229621d0b3467758321504a2e6477a932caaa WatchSource:0}: Error finding container 11fce052ec3570fb9b0c25f6f07229621d0b3467758321504a2e6477a932caaa: Status 404 returned error can't find the container with id 11fce052ec3570fb9b0c25f6f07229621d0b3467758321504a2e6477a932caaa Jul 15 14:09:56.235075 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.235042 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:09:56.235166 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.235124 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:09:56.235236 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.235171 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:09:56.703925 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.703891 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" event={"ID":"87cb802c-3122-4e4d-8504-0394abd7877a","Type":"ContainerStarted","Data":"89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655"} Jul 15 14:09:56.704124 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.703945 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" event={"ID":"87cb802c-3122-4e4d-8504-0394abd7877a","Type":"ContainerStarted","Data":"11fce052ec3570fb9b0c25f6f07229621d0b3467758321504a2e6477a932caaa"} Jul 15 14:09:56.726138 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:56.726079 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" podStartSLOduration=1.7260601530000002 podStartE2EDuration="1.726060153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:09:56.725087051 +0000 UTC m=+2438.242711251" watchObservedRunningTime="2026-07-15 14:09:56.726060153 +0000 UTC m=+2438.243684354" Jul 15 14:09:57.095894 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:57.095872 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:57.097870 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:57.097850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:57.708714 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:57.708681 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:09:57.709823 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:09:57.709798 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:10:03.475768 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:03.475729 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:10:11.781303 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:11.781268 2576 generic.go:356] "Generic (PLEG): container finished" podID="87cb802c-3122-4e4d-8504-0394abd7877a" containerID="89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655" exitCode=0 Jul 15 14:10:11.781758 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:11.781324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" event={"ID":"87cb802c-3122-4e4d-8504-0394abd7877a","Type":"ContainerDied","Data":"89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655"} Jul 15 14:10:11.781758 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:11.781717 2576 scope.go:117] "RemoveContainer" containerID="89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655" Jul 15 14:10:11.782717 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:11.782689 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:10:11.782784 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:11.782756 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:10:11.782827 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:11.782805 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:10:12.708731 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:12.708696 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:10:12.788414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:12.788365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" event={"ID":"87cb802c-3122-4e4d-8504-0394abd7877a","Type":"ContainerStarted","Data":"b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc"} Jul 15 14:10:13.094119 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:13.094030 2576 patch_prober.go:28] interesting pod/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 14:10:13.094119 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:13.094083 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 14:10:13.097055 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:13.097035 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:10:14.093784 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:14.093742 2576 patch_prober.go:28] interesting pod/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 14:10:14.094182 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:14.093802 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 14:10:15.093023 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:15.092987 2576 patch_prober.go:28] interesting pod/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 14:10:15.093258 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:15.093042 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 14:10:15.773545 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:15.773517 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:10:16.093690 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:16.093605 2576 patch_prober.go:28] interesting pod/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 14:10:16.093690 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:16.093663 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 14:10:17.097438 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:17.097384 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:10:17.098251 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:17.098231 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:10:17.098851 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:17.098830 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:10:32.476287 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.476251 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9"] Jul 15 14:10:32.479933 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.479916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.482813 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.482794 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2723e9\"/\"kube-root-ca.crt\"" Jul 15 14:10:32.482938 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.482794 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2723e9\"/\"openshift-service-ca.crt\"" Jul 15 14:10:32.482938 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.482794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2723e9\"/\"default-dockercfg-c96r6\"" Jul 15 14:10:32.482938 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.482794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2723e9\"/\"auth-test-model-2723e9-kserve-self-signed-certs\"" Jul 15 14:10:32.492235 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.492214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9"] Jul 15 14:10:32.564632 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.564596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af18d545-c98a-42bc-ab58-5095aa782574-tls-certs\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.564632 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.564635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-dshm\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.564887 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.564672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhl7\" (UniqueName: \"kubernetes.io/projected/af18d545-c98a-42bc-ab58-5095aa782574-kube-api-access-hnhl7\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.564887 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.564736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-tmp-dir\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.564887 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.564803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-home\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.565009 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.564928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-model-cache\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.665589 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.665555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af18d545-c98a-42bc-ab58-5095aa782574-tls-certs\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.665774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.665597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-dshm\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.665774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.665620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhl7\" (UniqueName: \"kubernetes.io/projected/af18d545-c98a-42bc-ab58-5095aa782574-kube-api-access-hnhl7\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.665774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.665648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-tmp-dir\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.665774 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.665676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-home\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.665936 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.665792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-model-cache\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.666122 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.666103 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-home\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.666184 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.666129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-tmp-dir\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.666245 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.666188 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-model-cache\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.667901 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.667876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-dshm\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.668136 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.668118 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af18d545-c98a-42bc-ab58-5095aa782574-tls-certs\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.676524 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.676500 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhl7\" (UniqueName: \"kubernetes.io/projected/af18d545-c98a-42bc-ab58-5095aa782574-kube-api-access-hnhl7\") pod \"auth-test-model-2723e9-kserve-5fd676c947-lcwg9\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.790820 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.790733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:32.924326 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:32.924292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9"] Jul 15 14:10:32.927316 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:10:32.927289 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf18d545_c98a_42bc_ab58_5095aa782574.slice/crio-3d31dfdbe0b1790f1f2218f36c1e331560696ff66c5749ecd60062b1127a4d8d WatchSource:0}: Error finding container 3d31dfdbe0b1790f1f2218f36c1e331560696ff66c5749ecd60062b1127a4d8d: Status 404 returned error can't find the container with id 3d31dfdbe0b1790f1f2218f36c1e331560696ff66c5749ecd60062b1127a4d8d Jul 15 14:10:33.895389 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:33.895355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" event={"ID":"af18d545-c98a-42bc-ab58-5095aa782574","Type":"ContainerStarted","Data":"101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87"} Jul 15 14:10:33.895806 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:33.895415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" event={"ID":"af18d545-c98a-42bc-ab58-5095aa782574","Type":"ContainerStarted","Data":"3d31dfdbe0b1790f1f2218f36c1e331560696ff66c5749ecd60062b1127a4d8d"} Jul 15 14:10:33.912900 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:33.912852 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" podStartSLOduration=1.91283518 podStartE2EDuration="1.91283518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:10:33.912792325 +0000 UTC m=+2475.430416526" watchObservedRunningTime="2026-07-15 14:10:33.91283518 +0000 UTC m=+2475.430459381" Jul 15 14:10:35.601985 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:35.601948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:10:42.791478 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:42.791435 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:42.791906 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:42.791517 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:42.804136 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:42.804108 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:10:42.949151 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:10:42.949125 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:11:11.880429 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.880333 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj"] Jul 15 14:11:11.883922 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.883901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:11.886483 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.886455 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"default-dockercfg-p5g97\"" Jul 15 14:11:11.886483 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.886475 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"auth-test-model-bfdd40-kserve-self-signed-certs\"" Jul 15 14:11:11.886622 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.886483 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"openshift-service-ca.crt\"" Jul 15 14:11:11.886622 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.886541 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"kube-root-ca.crt\"" Jul 15 14:11:11.895353 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.895333 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj"] Jul 15 14:11:11.932237 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.932213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d34806-fea9-4e81-911c-e9639ea0df61-tls-certs\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:11.932341 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.932267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-home\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:11.932341 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.932321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-tmp-dir\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:11.932443 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.932356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5knls\" (UniqueName: \"kubernetes.io/projected/f6d34806-fea9-4e81-911c-e9639ea0df61-kube-api-access-5knls\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:11.932443 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.932422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-dshm\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:11.932518 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:11.932472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-model-cache\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.033918 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.033879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-home\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.033918 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.033923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-tmp-dir\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.034126 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.034050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5knls\" (UniqueName: \"kubernetes.io/projected/f6d34806-fea9-4e81-911c-e9639ea0df61-kube-api-access-5knls\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.034126 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.034117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-dshm\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.034210 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.034168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-model-cache\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.034258 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.034237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d34806-fea9-4e81-911c-e9639ea0df61-tls-certs\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.034321 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.034290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-tmp-dir\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.034377 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.034334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-home\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.034614 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.034592 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-model-cache\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.036340 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.036320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-dshm\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.036626 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.036607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d34806-fea9-4e81-911c-e9639ea0df61-tls-certs\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.041782 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.041759 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5knls\" (UniqueName: \"kubernetes.io/projected/f6d34806-fea9-4e81-911c-e9639ea0df61-kube-api-access-5knls\") pod \"auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.193385 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.193281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:12.329898 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:12.329871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj"] Jul 15 14:11:12.332041 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:11:12.332017 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6d34806_fea9_4e81_911c_e9639ea0df61.slice/crio-e3084947cf64f48360ccf80499e6e248d9e1e8846bbf32292c01cb4aff6a51af WatchSource:0}: Error finding container e3084947cf64f48360ccf80499e6e248d9e1e8846bbf32292c01cb4aff6a51af: Status 404 returned error can't find the container with id e3084947cf64f48360ccf80499e6e248d9e1e8846bbf32292c01cb4aff6a51af Jul 15 14:11:13.083045 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:13.083011 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" event={"ID":"f6d34806-fea9-4e81-911c-e9639ea0df61","Type":"ContainerStarted","Data":"6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e"} Jul 15 14:11:13.083443 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:13.083054 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" event={"ID":"f6d34806-fea9-4e81-911c-e9639ea0df61","Type":"ContainerStarted","Data":"e3084947cf64f48360ccf80499e6e248d9e1e8846bbf32292c01cb4aff6a51af"} Jul 15 14:11:13.104471 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:13.104419 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" podStartSLOduration=2.104374452 podStartE2EDuration="2.104374452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:11:13.103208856 +0000 UTC m=+2514.620833055" watchObservedRunningTime="2026-07-15 14:11:13.104374452 +0000 UTC m=+2514.621998654" Jul 15 14:11:15.692758 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:15.692724 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:11:22.194260 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:22.194219 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:22.194710 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:22.194303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:22.206654 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:22.206630 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:23.150667 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:23.150640 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:11:29.975227 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:29.975192 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:11:47.285479 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:11:47.285445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:12:04.878890 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:12:04.878857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:12:22.485519 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:12:22.485484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:12:39.677109 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:12:39.677075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:12:56.486434 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:12:56.486334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:13:09.084664 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:09.084630 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9"] Jul 15 14:13:09.978495 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:09.978457 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj"] Jul 15 14:13:14.695531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:14.695496 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:13:17.838646 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.838585 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w"] Jul 15 14:13:17.842225 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.842208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.848942 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.848913 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2723e9\"/\"sub-test-model-2723e9-kserve-self-signed-certs\"" Jul 15 14:13:17.855420 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.855389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-dshm\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.855486 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.855441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tmp-dir\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.855486 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.855474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-model-cache\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.855555 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.855506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tls-certs\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.855593 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.855578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6dp\" (UniqueName: \"kubernetes.io/projected/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-kube-api-access-7s6dp\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.855627 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.855619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-home\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.872849 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.872826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w"] Jul 15 14:13:17.956050 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6dp\" (UniqueName: \"kubernetes.io/projected/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-kube-api-access-7s6dp\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956193 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-home\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956193 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-dshm\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956193 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tmp-dir\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956336 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-model-cache\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956336 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tls-certs\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956580 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956555 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-home\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956647 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tmp-dir\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.956688 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.956668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-model-cache\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.958412 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.958384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-dshm\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.958588 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.958570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tls-certs\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:17.965173 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:17.965150 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6dp\" (UniqueName: \"kubernetes.io/projected/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-kube-api-access-7s6dp\") pod \"sub-test-model-2723e9-kserve-5484bbd867-jql7w\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:18.151841 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:18.151813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:18.286924 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:18.286900 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w"] Jul 15 14:13:18.288800 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:13:18.288772 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0003ff86_4ce7_4bdf_b8a0_4f55130424ca.slice/crio-cec11e7f29c7fc7f45252866007768e6c4a866d1be7115170cf006f451cda808 WatchSource:0}: Error finding container cec11e7f29c7fc7f45252866007768e6c4a866d1be7115170cf006f451cda808: Status 404 returned error can't find the container with id cec11e7f29c7fc7f45252866007768e6c4a866d1be7115170cf006f451cda808 Jul 15 14:13:18.708956 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:18.708918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" event={"ID":"0003ff86-4ce7-4bdf-b8a0-4f55130424ca","Type":"ContainerStarted","Data":"0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538"} Jul 15 14:13:18.708956 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:18.708962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" event={"ID":"0003ff86-4ce7-4bdf-b8a0-4f55130424ca","Type":"ContainerStarted","Data":"cec11e7f29c7fc7f45252866007768e6c4a866d1be7115170cf006f451cda808"} Jul 15 14:13:18.729855 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:18.729680 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" podStartSLOduration=1.729653313 podStartE2EDuration="1.729653313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:13:18.728955594 +0000 UTC m=+2640.246579794" watchObservedRunningTime="2026-07-15 14:13:18.729653313 +0000 UTC m=+2640.247277515" Jul 15 14:13:24.104866 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.104803 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" podUID="af18d545-c98a-42bc-ab58-5095aa782574" containerName="main" containerID="cri-o://101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87" gracePeriod=45 Jul 15 14:13:24.354260 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.354234 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:13:24.406220 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-tmp-dir\") pod \"af18d545-c98a-42bc-ab58-5095aa782574\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " Jul 15 14:13:24.406348 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406241 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af18d545-c98a-42bc-ab58-5095aa782574-tls-certs\") pod \"af18d545-c98a-42bc-ab58-5095aa782574\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " Jul 15 14:13:24.406348 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406262 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-model-cache\") pod \"af18d545-c98a-42bc-ab58-5095aa782574\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " Jul 15 14:13:24.406348 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406299 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-home\") pod \"af18d545-c98a-42bc-ab58-5095aa782574\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " Jul 15 14:13:24.406500 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406347 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-dshm\") pod \"af18d545-c98a-42bc-ab58-5095aa782574\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " Jul 15 14:13:24.406500 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnhl7\" (UniqueName: \"kubernetes.io/projected/af18d545-c98a-42bc-ab58-5095aa782574-kube-api-access-hnhl7\") pod \"af18d545-c98a-42bc-ab58-5095aa782574\" (UID: \"af18d545-c98a-42bc-ab58-5095aa782574\") " Jul 15 14:13:24.406500 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406434 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af18d545-c98a-42bc-ab58-5095aa782574" (UID: "af18d545-c98a-42bc-ab58-5095aa782574"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:24.406657 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406534 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-model-cache" (OuterVolumeSpecName: "model-cache") pod "af18d545-c98a-42bc-ab58-5095aa782574" (UID: "af18d545-c98a-42bc-ab58-5095aa782574"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:24.406657 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-home" (OuterVolumeSpecName: "home") pod "af18d545-c98a-42bc-ab58-5095aa782574" (UID: "af18d545-c98a-42bc-ab58-5095aa782574"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:24.406780 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406760 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:24.406846 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406788 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:24.406846 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.406821 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:24.408455 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.408424 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af18d545-c98a-42bc-ab58-5095aa782574-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af18d545-c98a-42bc-ab58-5095aa782574" (UID: "af18d545-c98a-42bc-ab58-5095aa782574"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:13:24.408586 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.408453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-dshm" (OuterVolumeSpecName: "dshm") pod "af18d545-c98a-42bc-ab58-5095aa782574" (UID: "af18d545-c98a-42bc-ab58-5095aa782574"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:24.408586 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.408468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af18d545-c98a-42bc-ab58-5095aa782574-kube-api-access-hnhl7" (OuterVolumeSpecName: "kube-api-access-hnhl7") pod "af18d545-c98a-42bc-ab58-5095aa782574" (UID: "af18d545-c98a-42bc-ab58-5095aa782574"). InnerVolumeSpecName "kube-api-access-hnhl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:13:24.508305 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.508271 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af18d545-c98a-42bc-ab58-5095aa782574-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:24.508305 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.508300 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnhl7\" (UniqueName: \"kubernetes.io/projected/af18d545-c98a-42bc-ab58-5095aa782574-kube-api-access-hnhl7\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:24.508506 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.508314 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af18d545-c98a-42bc-ab58-5095aa782574-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:24.740772 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.740688 2576 generic.go:356] "Generic (PLEG): container finished" podID="af18d545-c98a-42bc-ab58-5095aa782574" containerID="101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87" exitCode=0 Jul 15 14:13:24.740772 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.740729 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" Jul 15 14:13:24.740772 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.740746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" event={"ID":"af18d545-c98a-42bc-ab58-5095aa782574","Type":"ContainerDied","Data":"101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87"} Jul 15 14:13:24.741000 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.740785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9" event={"ID":"af18d545-c98a-42bc-ab58-5095aa782574","Type":"ContainerDied","Data":"3d31dfdbe0b1790f1f2218f36c1e331560696ff66c5749ecd60062b1127a4d8d"} Jul 15 14:13:24.741000 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.740803 2576 scope.go:117] "RemoveContainer" containerID="101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87" Jul 15 14:13:24.749841 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.749822 2576 scope.go:117] "RemoveContainer" containerID="101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87" Jul 15 14:13:24.750127 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:13:24.750108 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87\": container with ID starting with 101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87 not found: ID does not exist" containerID="101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87" Jul 15 14:13:24.750194 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.750133 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87"} err="failed to get container status \"101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87\": rpc error: code = NotFound desc = could not find container \"101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87\": container with ID starting with 101e03753bc34e1a2bcb091ac90670252442aaa4337d69320ed924288c433c87 not found: ID does not exist" Jul 15 14:13:24.767220 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.767196 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9"] Jul 15 14:13:24.771346 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.771324 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/auth-test-model-2723e9-kserve-5fd676c947-lcwg9"] Jul 15 14:13:24.999159 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:24.999069 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" podUID="f6d34806-fea9-4e81-911c-e9639ea0df61" containerName="main" containerID="cri-o://6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e" gracePeriod=45 Jul 15 14:13:25.096816 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.096784 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af18d545-c98a-42bc-ab58-5095aa782574" path="/var/lib/kubelet/pods/af18d545-c98a-42bc-ab58-5095aa782574/volumes" Jul 15 14:13:25.255629 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.255567 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:13:25.315903 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.315874 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5knls\" (UniqueName: \"kubernetes.io/projected/f6d34806-fea9-4e81-911c-e9639ea0df61-kube-api-access-5knls\") pod \"f6d34806-fea9-4e81-911c-e9639ea0df61\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " Jul 15 14:13:25.316048 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.315915 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-dshm\") pod \"f6d34806-fea9-4e81-911c-e9639ea0df61\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " Jul 15 14:13:25.316048 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.315953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d34806-fea9-4e81-911c-e9639ea0df61-tls-certs\") pod \"f6d34806-fea9-4e81-911c-e9639ea0df61\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " Jul 15 14:13:25.316048 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.315972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-home\") pod \"f6d34806-fea9-4e81-911c-e9639ea0df61\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " Jul 15 14:13:25.316048 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.315998 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-model-cache\") pod \"f6d34806-fea9-4e81-911c-e9639ea0df61\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " Jul 15 14:13:25.316048 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.316023 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-tmp-dir\") pod \"f6d34806-fea9-4e81-911c-e9639ea0df61\" (UID: \"f6d34806-fea9-4e81-911c-e9639ea0df61\") " Jul 15 14:13:25.316304 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.316262 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-home" (OuterVolumeSpecName: "home") pod "f6d34806-fea9-4e81-911c-e9639ea0df61" (UID: "f6d34806-fea9-4e81-911c-e9639ea0df61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:25.316415 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.316349 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6d34806-fea9-4e81-911c-e9639ea0df61" (UID: "f6d34806-fea9-4e81-911c-e9639ea0df61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:25.316528 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.316437 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6d34806-fea9-4e81-911c-e9639ea0df61" (UID: "f6d34806-fea9-4e81-911c-e9639ea0df61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:25.318186 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.318157 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d34806-fea9-4e81-911c-e9639ea0df61-kube-api-access-5knls" (OuterVolumeSpecName: "kube-api-access-5knls") pod "f6d34806-fea9-4e81-911c-e9639ea0df61" (UID: "f6d34806-fea9-4e81-911c-e9639ea0df61"). InnerVolumeSpecName "kube-api-access-5knls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:13:25.318186 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.318166 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-dshm" (OuterVolumeSpecName: "dshm") pod "f6d34806-fea9-4e81-911c-e9639ea0df61" (UID: "f6d34806-fea9-4e81-911c-e9639ea0df61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:13:25.318311 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.318213 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d34806-fea9-4e81-911c-e9639ea0df61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6d34806-fea9-4e81-911c-e9639ea0df61" (UID: "f6d34806-fea9-4e81-911c-e9639ea0df61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:13:25.417346 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.417313 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5knls\" (UniqueName: \"kubernetes.io/projected/f6d34806-fea9-4e81-911c-e9639ea0df61-kube-api-access-5knls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:25.417346 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.417340 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:25.417553 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.417352 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d34806-fea9-4e81-911c-e9639ea0df61-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:25.417553 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.417366 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:25.417553 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.417376 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:25.417553 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.417387 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d34806-fea9-4e81-911c-e9639ea0df61-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:13:25.747472 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.747431 2576 generic.go:356] "Generic (PLEG): container finished" podID="f6d34806-fea9-4e81-911c-e9639ea0df61" containerID="6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e" exitCode=0 Jul 15 14:13:25.747472 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.747465 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" Jul 15 14:13:25.747683 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.747501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" event={"ID":"f6d34806-fea9-4e81-911c-e9639ea0df61","Type":"ContainerDied","Data":"6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e"} Jul 15 14:13:25.747683 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.747542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj" event={"ID":"f6d34806-fea9-4e81-911c-e9639ea0df61","Type":"ContainerDied","Data":"e3084947cf64f48360ccf80499e6e248d9e1e8846bbf32292c01cb4aff6a51af"} Jul 15 14:13:25.747683 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.747561 2576 scope.go:117] "RemoveContainer" containerID="6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e" Jul 15 14:13:25.756889 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.756869 2576 scope.go:117] "RemoveContainer" containerID="6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e" Jul 15 14:13:25.757130 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:13:25.757112 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e\": container with ID starting with 6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e not found: ID does not exist" containerID="6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e" Jul 15 14:13:25.757175 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.757139 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e"} err="failed to get container status \"6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e\": rpc error: code = NotFound desc = could not find container \"6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e\": container with ID starting with 6ed9f49dab8f78e8017b243ba13f24819b1e690a7847351c3116cd093235ed4e not found: ID does not exist" Jul 15 14:13:25.774961 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.774933 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj"] Jul 15 14:13:25.782512 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:25.782489 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/auth-test-model-bfdd40-kserve-6dbcb848c-r8zmj"] Jul 15 14:13:27.096839 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:27.096802 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d34806-fea9-4e81-911c-e9639ea0df61" path="/var/lib/kubelet/pods/f6d34806-fea9-4e81-911c-e9639ea0df61/volumes" Jul 15 14:13:28.152344 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:28.152307 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:28.152816 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:28.152361 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:28.164929 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:28.164902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:28.773538 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:28.773508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:13:31.903343 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:13:31.903307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:14:13.661524 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.661439 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz"] Jul 15 14:14:13.661974 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.661833 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af18d545-c98a-42bc-ab58-5095aa782574" containerName="main" Jul 15 14:14:13.661974 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.661848 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="af18d545-c98a-42bc-ab58-5095aa782574" containerName="main" Jul 15 14:14:13.661974 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.661863 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d34806-fea9-4e81-911c-e9639ea0df61" containerName="main" Jul 15 14:14:13.661974 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.661868 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d34806-fea9-4e81-911c-e9639ea0df61" containerName="main" Jul 15 14:14:13.661974 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.661941 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="af18d545-c98a-42bc-ab58-5095aa782574" containerName="main" Jul 15 14:14:13.661974 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.661948 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d34806-fea9-4e81-911c-e9639ea0df61" containerName="main" Jul 15 14:14:13.664857 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.664840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.669930 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.669908 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"kube-root-ca.crt\"" Jul 15 14:14:13.670494 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.670471 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"openshift-service-ca.crt\"" Jul 15 14:14:13.670617 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.670476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"sub-test-model-bfdd40-kserve-self-signed-certs\"" Jul 15 14:14:13.670617 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.670535 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"default-dockercfg-p5g97\"" Jul 15 14:14:13.697535 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.697510 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz"] Jul 15 14:14:13.773877 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.773838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24719f3-a600-4db2-84d1-8ff034eb629f-tls-certs\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.773877 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.773878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-model-cache\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.774090 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.773971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-tmp-dir\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.774090 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.774013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flp9k\" (UniqueName: \"kubernetes.io/projected/b24719f3-a600-4db2-84d1-8ff034eb629f-kube-api-access-flp9k\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.774090 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.774050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-dshm\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.774188 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.774119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-home\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.874663 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.874622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flp9k\" (UniqueName: \"kubernetes.io/projected/b24719f3-a600-4db2-84d1-8ff034eb629f-kube-api-access-flp9k\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.874851 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.874676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-dshm\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.874851 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.874712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-home\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.874851 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.874752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24719f3-a600-4db2-84d1-8ff034eb629f-tls-certs\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.874851 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.874769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-model-cache\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.874851 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.874835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-tmp-dir\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.875155 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.875127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-home\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.875302 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.875253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-tmp-dir\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.875302 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.875279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-model-cache\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.877103 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.877079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-dshm\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.877210 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.877192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24719f3-a600-4db2-84d1-8ff034eb629f-tls-certs\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.883280 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.883259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flp9k\" (UniqueName: \"kubernetes.io/projected/b24719f3-a600-4db2-84d1-8ff034eb629f-kube-api-access-flp9k\") pod \"sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:13.973915 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:13.973814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:14.110752 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:14.110726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz"] Jul 15 14:14:14.112921 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:14:14.112893 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb24719f3_a600_4db2_84d1_8ff034eb629f.slice/crio-d0df25e9a78a68a6527cc79c55aa1d6d7081ec8e903130668b44f0ca646d7b9f WatchSource:0}: Error finding container d0df25e9a78a68a6527cc79c55aa1d6d7081ec8e903130668b44f0ca646d7b9f: Status 404 returned error can't find the container with id d0df25e9a78a68a6527cc79c55aa1d6d7081ec8e903130668b44f0ca646d7b9f Jul 15 14:14:15.003958 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:15.003917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" event={"ID":"b24719f3-a600-4db2-84d1-8ff034eb629f","Type":"ContainerStarted","Data":"fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4"} Jul 15 14:14:15.003958 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:15.003963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" event={"ID":"b24719f3-a600-4db2-84d1-8ff034eb629f","Type":"ContainerStarted","Data":"d0df25e9a78a68a6527cc79c55aa1d6d7081ec8e903130668b44f0ca646d7b9f"} Jul 15 14:14:15.023617 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:15.023570 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" podStartSLOduration=2.023554747 podStartE2EDuration="2.023554747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:14:15.022082302 +0000 UTC m=+2696.539706514" watchObservedRunningTime="2026-07-15 14:14:15.023554747 +0000 UTC m=+2696.541178946" Jul 15 14:14:18.407087 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:18.407052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:14:19.147263 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:19.147235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 14:14:19.165344 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:19.165317 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rps9b_944ce9ec-3ae4-431a-8f68-c6c720fc7613/ovn-acl-logging/0.log" Jul 15 14:14:23.974749 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:23.974711 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:23.974749 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:23.974762 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:23.987493 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:23.987467 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:24.058902 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:24.058870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:14:42.486999 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:14:42.486955 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:15:00.194279 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:00.194235 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:15:05.989653 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:05.989609 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w"] Jul 15 14:15:06.989707 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:06.989660 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz"] Jul 15 14:15:20.483372 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:20.483336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:15:21.010188 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.010147 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" podUID="0003ff86-4ce7-4bdf-b8a0-4f55130424ca" containerName="main" containerID="cri-o://0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538" gracePeriod=45 Jul 15 14:15:21.265606 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.265539 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:15:21.329278 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.329242 2576 generic.go:356] "Generic (PLEG): container finished" podID="0003ff86-4ce7-4bdf-b8a0-4f55130424ca" containerID="0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538" exitCode=0 Jul 15 14:15:21.329459 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.329284 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" Jul 15 14:15:21.329459 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.329286 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" event={"ID":"0003ff86-4ce7-4bdf-b8a0-4f55130424ca","Type":"ContainerDied","Data":"0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538"} Jul 15 14:15:21.329459 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.329441 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w" event={"ID":"0003ff86-4ce7-4bdf-b8a0-4f55130424ca","Type":"ContainerDied","Data":"cec11e7f29c7fc7f45252866007768e6c4a866d1be7115170cf006f451cda808"} Jul 15 14:15:21.329585 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.329475 2576 scope.go:117] "RemoveContainer" containerID="0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538" Jul 15 14:15:21.333313 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333293 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-home\") pod \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " Jul 15 14:15:21.333437 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-dshm\") pod \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " Jul 15 14:15:21.333437 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tls-certs\") pod \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " Jul 15 14:15:21.333570 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333488 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s6dp\" (UniqueName: \"kubernetes.io/projected/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-kube-api-access-7s6dp\") pod \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " Jul 15 14:15:21.333570 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333517 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-model-cache\") pod \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " Jul 15 14:15:21.333679 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333578 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-home" (OuterVolumeSpecName: "home") pod "0003ff86-4ce7-4bdf-b8a0-4f55130424ca" (UID: "0003ff86-4ce7-4bdf-b8a0-4f55130424ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:21.333679 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tmp-dir\") pod \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\" (UID: \"0003ff86-4ce7-4bdf-b8a0-4f55130424ca\") " Jul 15 14:15:21.333871 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333807 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "0003ff86-4ce7-4bdf-b8a0-4f55130424ca" (UID: "0003ff86-4ce7-4bdf-b8a0-4f55130424ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:21.334017 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.333996 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0003ff86-4ce7-4bdf-b8a0-4f55130424ca" (UID: "0003ff86-4ce7-4bdf-b8a0-4f55130424ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:21.334095 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.334056 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:21.334095 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.334076 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:21.335633 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.335606 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0003ff86-4ce7-4bdf-b8a0-4f55130424ca" (UID: "0003ff86-4ce7-4bdf-b8a0-4f55130424ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:15:21.335903 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.335882 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-dshm" (OuterVolumeSpecName: "dshm") pod "0003ff86-4ce7-4bdf-b8a0-4f55130424ca" (UID: "0003ff86-4ce7-4bdf-b8a0-4f55130424ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:21.335988 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.335917 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-kube-api-access-7s6dp" (OuterVolumeSpecName: "kube-api-access-7s6dp") pod "0003ff86-4ce7-4bdf-b8a0-4f55130424ca" (UID: "0003ff86-4ce7-4bdf-b8a0-4f55130424ca"). InnerVolumeSpecName "kube-api-access-7s6dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:15:21.338321 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.338306 2576 scope.go:117] "RemoveContainer" containerID="0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538" Jul 15 14:15:21.338641 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:15:21.338620 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538\": container with ID starting with 0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538 not found: ID does not exist" containerID="0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538" Jul 15 14:15:21.338716 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.338646 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538"} err="failed to get container status \"0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538\": rpc error: code = NotFound desc = could not find container \"0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538\": container with ID starting with 0696f732e257bb7285c1f57fcb4dd590186b8d69bdefb1f7826871e698c09538 not found: ID does not exist" Jul 15 14:15:21.435216 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.435186 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:21.435216 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.435209 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:21.435216 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.435219 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:21.435464 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.435228 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s6dp\" (UniqueName: \"kubernetes.io/projected/0003ff86-4ce7-4bdf-b8a0-4f55130424ca-kube-api-access-7s6dp\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:21.652101 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.652057 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w"] Jul 15 14:15:21.659059 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:21.659027 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2723e9/sub-test-model-2723e9-kserve-5484bbd867-jql7w"] Jul 15 14:15:22.010075 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.009984 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" podUID="b24719f3-a600-4db2-84d1-8ff034eb629f" containerName="main" containerID="cri-o://fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4" gracePeriod=45 Jul 15 14:15:22.252976 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.252948 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:15:22.335824 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.335734 2576 generic.go:356] "Generic (PLEG): container finished" podID="b24719f3-a600-4db2-84d1-8ff034eb629f" containerID="fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4" exitCode=0 Jul 15 14:15:22.335824 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.335780 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" Jul 15 14:15:22.335824 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.335805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" event={"ID":"b24719f3-a600-4db2-84d1-8ff034eb629f","Type":"ContainerDied","Data":"fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4"} Jul 15 14:15:22.336061 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.335837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz" event={"ID":"b24719f3-a600-4db2-84d1-8ff034eb629f","Type":"ContainerDied","Data":"d0df25e9a78a68a6527cc79c55aa1d6d7081ec8e903130668b44f0ca646d7b9f"} Jul 15 14:15:22.336061 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.335854 2576 scope.go:117] "RemoveContainer" containerID="fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4" Jul 15 14:15:22.343780 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.343756 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24719f3-a600-4db2-84d1-8ff034eb629f-tls-certs\") pod \"b24719f3-a600-4db2-84d1-8ff034eb629f\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " Jul 15 14:15:22.343780 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.343783 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-home\") pod \"b24719f3-a600-4db2-84d1-8ff034eb629f\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " Jul 15 14:15:22.343966 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.343853 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-model-cache\") pod \"b24719f3-a600-4db2-84d1-8ff034eb629f\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " Jul 15 14:15:22.343966 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.343896 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-dshm\") pod \"b24719f3-a600-4db2-84d1-8ff034eb629f\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " Jul 15 14:15:22.343966 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.343951 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-tmp-dir\") pod \"b24719f3-a600-4db2-84d1-8ff034eb629f\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " Jul 15 14:15:22.344414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.343992 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flp9k\" (UniqueName: \"kubernetes.io/projected/b24719f3-a600-4db2-84d1-8ff034eb629f-kube-api-access-flp9k\") pod \"b24719f3-a600-4db2-84d1-8ff034eb629f\" (UID: \"b24719f3-a600-4db2-84d1-8ff034eb629f\") " Jul 15 14:15:22.344414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.344160 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-home" (OuterVolumeSpecName: "home") pod "b24719f3-a600-4db2-84d1-8ff034eb629f" (UID: "b24719f3-a600-4db2-84d1-8ff034eb629f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:22.344414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.344162 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b24719f3-a600-4db2-84d1-8ff034eb629f" (UID: "b24719f3-a600-4db2-84d1-8ff034eb629f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:22.344414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.343994 2576 scope.go:117] "RemoveContainer" containerID="fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4" Jul 15 14:15:22.344414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.344206 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b24719f3-a600-4db2-84d1-8ff034eb629f" (UID: "b24719f3-a600-4db2-84d1-8ff034eb629f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:22.344414 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.344379 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:22.344771 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.344421 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:22.344771 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.344437 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:22.344771 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:15:22.344502 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4\": container with ID starting with fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4 not found: ID does not exist" containerID="fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4" Jul 15 14:15:22.344771 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.344537 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4"} err="failed to get container status \"fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4\": rpc error: code = NotFound desc = could not find container \"fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4\": container with ID starting with fd6bb25173251e5d596a86094592a124ec71b26c4f76e378ae0d201933bf8df4 not found: ID does not exist" Jul 15 14:15:22.345827 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.345805 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24719f3-a600-4db2-84d1-8ff034eb629f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b24719f3-a600-4db2-84d1-8ff034eb629f" (UID: "b24719f3-a600-4db2-84d1-8ff034eb629f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:15:22.345927 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.345887 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-dshm" (OuterVolumeSpecName: "dshm") pod "b24719f3-a600-4db2-84d1-8ff034eb629f" (UID: "b24719f3-a600-4db2-84d1-8ff034eb629f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:15:22.346153 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.346136 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24719f3-a600-4db2-84d1-8ff034eb629f-kube-api-access-flp9k" (OuterVolumeSpecName: "kube-api-access-flp9k") pod "b24719f3-a600-4db2-84d1-8ff034eb629f" (UID: "b24719f3-a600-4db2-84d1-8ff034eb629f"). InnerVolumeSpecName "kube-api-access-flp9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:15:22.445582 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.445547 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flp9k\" (UniqueName: \"kubernetes.io/projected/b24719f3-a600-4db2-84d1-8ff034eb629f-kube-api-access-flp9k\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:22.445582 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.445578 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b24719f3-a600-4db2-84d1-8ff034eb629f-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:22.445582 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.445593 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b24719f3-a600-4db2-84d1-8ff034eb629f-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:15:22.658927 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.658889 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz"] Jul 15 14:15:22.660821 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:22.660802 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/sub-test-model-bfdd40-kserve-575d6c9c59-p6kxz"] Jul 15 14:15:23.096117 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:23.096090 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0003ff86-4ce7-4bdf-b8a0-4f55130424ca" path="/var/lib/kubelet/pods/0003ff86-4ce7-4bdf-b8a0-4f55130424ca/volumes" Jul 15 14:15:23.096531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:23.096517 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24719f3-a600-4db2-84d1-8ff034eb629f" path="/var/lib/kubelet/pods/b24719f3-a600-4db2-84d1-8ff034eb629f/volumes" Jul 15 14:15:59.181816 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.181731 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt"] Jul 15 14:15:59.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.182332 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24719f3-a600-4db2-84d1-8ff034eb629f" containerName="main" Jul 15 14:15:59.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.182350 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24719f3-a600-4db2-84d1-8ff034eb629f" containerName="main" Jul 15 14:15:59.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.182369 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0003ff86-4ce7-4bdf-b8a0-4f55130424ca" containerName="main" Jul 15 14:15:59.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.182377 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0003ff86-4ce7-4bdf-b8a0-4f55130424ca" containerName="main" Jul 15 14:15:59.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.182493 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24719f3-a600-4db2-84d1-8ff034eb629f" containerName="main" Jul 15 14:15:59.182531 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.182505 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0003ff86-4ce7-4bdf-b8a0-4f55130424ca" containerName="main" Jul 15 14:15:59.186594 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.186573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.190603 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.190570 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"kube-root-ca.crt\"" Jul 15 14:15:59.190603 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.190578 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"rate-test-model-bfdd40-kserve-self-signed-certs\"" Jul 15 14:15:59.190603 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.190570 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"openshift-service-ca.crt\"" Jul 15 14:15:59.190847 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.190712 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bfdd40\"/\"default-dockercfg-p5g97\"" Jul 15 14:15:59.194002 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.193976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-model-cache\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.194202 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.194183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-tmp-dir\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.194326 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.194310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b1e138-712f-4f55-8fe3-289411b1235c-tls-certs\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.194552 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.194523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtzp\" (UniqueName: \"kubernetes.io/projected/c1b1e138-712f-4f55-8fe3-289411b1235c-kube-api-access-rwtzp\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.194684 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.194623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-home\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.194684 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.194664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-dshm\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.196737 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.196716 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt"] Jul 15 14:15:59.295315 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-model-cache\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295519 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-tmp-dir\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295519 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b1e138-712f-4f55-8fe3-289411b1235c-tls-certs\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295519 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtzp\" (UniqueName: \"kubernetes.io/projected/c1b1e138-712f-4f55-8fe3-289411b1235c-kube-api-access-rwtzp\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295519 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-home\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295519 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-dshm\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295800 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-tmp-dir\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295862 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295803 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-model-cache\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.295926 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.295862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-home\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.298268 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.298239 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-dshm\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.298438 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.298416 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b1e138-712f-4f55-8fe3-289411b1235c-tls-certs\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.305837 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.305812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtzp\" (UniqueName: \"kubernetes.io/projected/c1b1e138-712f-4f55-8fe3-289411b1235c-kube-api-access-rwtzp\") pod \"rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.496926 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.496826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:15:59.637094 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:15:59.637065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt"] Jul 15 14:15:59.638912 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:15:59.638883 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1b1e138_712f_4f55_8fe3_289411b1235c.slice/crio-904123df4671e94e10cefbd2a84ab10c771c4e0bed74682658b23874e85ba541 WatchSource:0}: Error finding container 904123df4671e94e10cefbd2a84ab10c771c4e0bed74682658b23874e85ba541: Status 404 returned error can't find the container with id 904123df4671e94e10cefbd2a84ab10c771c4e0bed74682658b23874e85ba541 Jul 15 14:16:00.522057 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:00.522016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" event={"ID":"c1b1e138-712f-4f55-8fe3-289411b1235c","Type":"ContainerStarted","Data":"b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f"} Jul 15 14:16:00.522467 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:00.522068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" event={"ID":"c1b1e138-712f-4f55-8fe3-289411b1235c","Type":"ContainerStarted","Data":"904123df4671e94e10cefbd2a84ab10c771c4e0bed74682658b23874e85ba541"} Jul 15 14:16:00.540924 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:00.540870 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" podStartSLOduration=1.540854279 podStartE2EDuration="1.540854279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:16:00.538905815 +0000 UTC m=+2802.056530032" watchObservedRunningTime="2026-07-15 14:16:00.540854279 +0000 UTC m=+2802.058478478" Jul 15 14:16:03.490561 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:03.490524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:16:09.497821 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:09.497787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:16:09.498326 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:09.497874 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:16:09.510373 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:09.510347 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:16:09.575730 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:09.575705 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:16:28.178475 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:28.178442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:16:43.673417 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:43.673356 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:16:50.866068 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:50.866036 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt"] Jul 15 14:16:51.995714 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:51.995679 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-139915260738864"] Jul 15 14:16:51.999442 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:51.999424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260738864" Jul 15 14:16:52.002522 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.002496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wljdh\"" Jul 15 14:16:52.011585 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.011558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915260738864"] Jul 15 14:16:52.109937 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.109904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rg2f\" (UniqueName: \"kubernetes.io/projected/9c964326-c51c-46c3-9e3a-c6403c2b4422-kube-api-access-7rg2f\") pod \"test-curl-7853-139915260738864\" (UID: \"9c964326-c51c-46c3-9e3a-c6403c2b4422\") " pod="opendatahub/test-curl-7853-139915260738864" Jul 15 14:16:52.211209 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.211163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rg2f\" (UniqueName: \"kubernetes.io/projected/9c964326-c51c-46c3-9e3a-c6403c2b4422-kube-api-access-7rg2f\") pod \"test-curl-7853-139915260738864\" (UID: \"9c964326-c51c-46c3-9e3a-c6403c2b4422\") " pod="opendatahub/test-curl-7853-139915260738864" Jul 15 14:16:52.219751 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.219724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rg2f\" (UniqueName: \"kubernetes.io/projected/9c964326-c51c-46c3-9e3a-c6403c2b4422-kube-api-access-7rg2f\") pod \"test-curl-7853-139915260738864\" (UID: \"9c964326-c51c-46c3-9e3a-c6403c2b4422\") " pod="opendatahub/test-curl-7853-139915260738864" Jul 15 14:16:52.315886 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.315810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260738864" Jul 15 14:16:52.445237 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.445217 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915260738864"] Jul 15 14:16:52.447642 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:16:52.447614 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c964326_c51c_46c3_9e3a_c6403c2b4422.slice/crio-d6b1227e17fb48da182669bfdf41a44ea44734329b3f571586c77fe0f487ee20 WatchSource:0}: Error finding container d6b1227e17fb48da182669bfdf41a44ea44734329b3f571586c77fe0f487ee20: Status 404 returned error can't find the container with id d6b1227e17fb48da182669bfdf41a44ea44734329b3f571586c77fe0f487ee20 Jul 15 14:16:52.449007 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.448989 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 14:16:52.780284 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:52.780243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915260738864" event={"ID":"9c964326-c51c-46c3-9e3a-c6403c2b4422","Type":"ContainerStarted","Data":"d6b1227e17fb48da182669bfdf41a44ea44734329b3f571586c77fe0f487ee20"} Jul 15 14:16:53.786714 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:53.786675 2576 generic.go:356] "Generic (PLEG): container finished" podID="9c964326-c51c-46c3-9e3a-c6403c2b4422" containerID="6ad4c825af0536ab6deb3a0a3c4fce5f0a4bca5669b975bf09ae2be7cde7655f" exitCode=0 Jul 15 14:16:53.787110 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:53.786756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915260738864" event={"ID":"9c964326-c51c-46c3-9e3a-c6403c2b4422","Type":"ContainerDied","Data":"6ad4c825af0536ab6deb3a0a3c4fce5f0a4bca5669b975bf09ae2be7cde7655f"} Jul 15 14:16:54.924413 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:54.924378 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260738864" Jul 15 14:16:54.963101 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:54.963072 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7853-139915260738864_9c964326-c51c-46c3-9e3a-c6403c2b4422/test-curl-7853-139915260738864/0.log" Jul 15 14:16:54.991413 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:54.991371 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-139915260738864"] Jul 15 14:16:54.993424 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:54.993391 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-139915260738864"] Jul 15 14:16:55.039746 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:55.039719 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rg2f\" (UniqueName: \"kubernetes.io/projected/9c964326-c51c-46c3-9e3a-c6403c2b4422-kube-api-access-7rg2f\") pod \"9c964326-c51c-46c3-9e3a-c6403c2b4422\" (UID: \"9c964326-c51c-46c3-9e3a-c6403c2b4422\") " Jul 15 14:16:55.041839 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:55.041820 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c964326-c51c-46c3-9e3a-c6403c2b4422-kube-api-access-7rg2f" (OuterVolumeSpecName: "kube-api-access-7rg2f") pod "9c964326-c51c-46c3-9e3a-c6403c2b4422" (UID: "9c964326-c51c-46c3-9e3a-c6403c2b4422"). InnerVolumeSpecName "kube-api-access-7rg2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:16:55.096334 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:55.096301 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c964326-c51c-46c3-9e3a-c6403c2b4422" path="/var/lib/kubelet/pods/9c964326-c51c-46c3-9e3a-c6403c2b4422/volumes" Jul 15 14:16:55.140821 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:55.140793 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rg2f\" (UniqueName: \"kubernetes.io/projected/9c964326-c51c-46c3-9e3a-c6403c2b4422-kube-api-access-7rg2f\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:16:55.798467 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:55.798443 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260738864" Jul 15 14:16:55.798611 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:55.798445 2576 scope.go:117] "RemoveContainer" containerID="6ad4c825af0536ab6deb3a0a3c4fce5f0a4bca5669b975bf09ae2be7cde7655f" Jul 15 14:16:58.733805 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.733766 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:16:58.734206 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.734188 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c964326-c51c-46c3-9e3a-c6403c2b4422" containerName="test-curl-7853-139915260738864" Jul 15 14:16:58.734252 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.734208 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c964326-c51c-46c3-9e3a-c6403c2b4422" containerName="test-curl-7853-139915260738864" Jul 15 14:16:58.734290 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.734269 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c964326-c51c-46c3-9e3a-c6403c2b4422" containerName="test-curl-7853-139915260738864" Jul 15 14:16:58.738986 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.738970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:16:58.741069 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.741048 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wljdh\"" Jul 15 14:16:58.743647 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.743626 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:16:58.878456 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.878417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxt8\" (UniqueName: \"kubernetes.io/projected/0e2c5726-360b-4f8d-acb4-02a44475c077-kube-api-access-bbxt8\") pod \"test-curl-7853-139915260888816\" (UID: \"0e2c5726-360b-4f8d-acb4-02a44475c077\") " pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:16:58.979858 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.979821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxt8\" (UniqueName: \"kubernetes.io/projected/0e2c5726-360b-4f8d-acb4-02a44475c077-kube-api-access-bbxt8\") pod \"test-curl-7853-139915260888816\" (UID: \"0e2c5726-360b-4f8d-acb4-02a44475c077\") " pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:16:58.987896 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:58.987827 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxt8\" (UniqueName: \"kubernetes.io/projected/0e2c5726-360b-4f8d-acb4-02a44475c077-kube-api-access-bbxt8\") pod \"test-curl-7853-139915260888816\" (UID: \"0e2c5726-360b-4f8d-acb4-02a44475c077\") " pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:16:59.049361 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:59.049329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:16:59.177174 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:59.177142 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:16:59.180186 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:16:59.180149 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e2c5726_360b_4f8d_acb4_02a44475c077.slice/crio-e71b78488db9c4e0c173d66bb624b1b42f4cb9014c4a4ae3ef98e69399122dce WatchSource:0}: Error finding container e71b78488db9c4e0c173d66bb624b1b42f4cb9014c4a4ae3ef98e69399122dce: Status 404 returned error can't find the container with id e71b78488db9c4e0c173d66bb624b1b42f4cb9014c4a4ae3ef98e69399122dce Jul 15 14:16:59.820151 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:59.820058 2576 generic.go:356] "Generic (PLEG): container finished" podID="0e2c5726-360b-4f8d-acb4-02a44475c077" containerID="2f5cd07cf7501b26d1562d9f23232774de79b5f115bd3d7856f0c9f0c044d77c" exitCode=0 Jul 15 14:16:59.820151 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:59.820118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915260888816" event={"ID":"0e2c5726-360b-4f8d-acb4-02a44475c077","Type":"ContainerDied","Data":"2f5cd07cf7501b26d1562d9f23232774de79b5f115bd3d7856f0c9f0c044d77c"} Jul 15 14:16:59.820579 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:16:59.820154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915260888816" event={"ID":"0e2c5726-360b-4f8d-acb4-02a44475c077","Type":"ContainerStarted","Data":"e71b78488db9c4e0c173d66bb624b1b42f4cb9014c4a4ae3ef98e69399122dce"} Jul 15 14:17:00.951922 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:00.951892 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:00.974106 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:00.974083 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7853-139915260888816_0e2c5726-360b-4f8d-acb4-02a44475c077/test-curl-7853-139915260888816/0.log" Jul 15 14:17:01.000277 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.000247 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:17:01.004020 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.003998 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:17:01.098868 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.098836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbxt8\" (UniqueName: \"kubernetes.io/projected/0e2c5726-360b-4f8d-acb4-02a44475c077-kube-api-access-bbxt8\") pod \"0e2c5726-360b-4f8d-acb4-02a44475c077\" (UID: \"0e2c5726-360b-4f8d-acb4-02a44475c077\") " Jul 15 14:17:01.100773 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.100752 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2c5726-360b-4f8d-acb4-02a44475c077-kube-api-access-bbxt8" (OuterVolumeSpecName: "kube-api-access-bbxt8") pod "0e2c5726-360b-4f8d-acb4-02a44475c077" (UID: "0e2c5726-360b-4f8d-acb4-02a44475c077"). InnerVolumeSpecName "kube-api-access-bbxt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:17:01.127177 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.127149 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:17:01.127540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.127527 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2c5726-360b-4f8d-acb4-02a44475c077" containerName="test-curl-7853-139915260888816" Jul 15 14:17:01.127600 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.127541 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2c5726-360b-4f8d-acb4-02a44475c077" containerName="test-curl-7853-139915260888816" Jul 15 14:17:01.127636 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.127618 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2c5726-360b-4f8d-acb4-02a44475c077" containerName="test-curl-7853-139915260888816" Jul 15 14:17:01.130481 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.130465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:01.135696 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.135674 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:17:01.199556 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.199528 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbxt8\" (UniqueName: \"kubernetes.io/projected/0e2c5726-360b-4f8d-acb4-02a44475c077-kube-api-access-bbxt8\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:01.300616 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.300587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg59n\" (UniqueName: \"kubernetes.io/projected/54aecf37-b3e7-40f6-9a18-bc0c45d471bd-kube-api-access-gg59n\") pod \"test-curl-7853-139915260888816\" (UID: \"54aecf37-b3e7-40f6-9a18-bc0c45d471bd\") " pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:01.401989 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.401914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg59n\" (UniqueName: \"kubernetes.io/projected/54aecf37-b3e7-40f6-9a18-bc0c45d471bd-kube-api-access-gg59n\") pod \"test-curl-7853-139915260888816\" (UID: \"54aecf37-b3e7-40f6-9a18-bc0c45d471bd\") " pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:01.409455 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.409433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg59n\" (UniqueName: \"kubernetes.io/projected/54aecf37-b3e7-40f6-9a18-bc0c45d471bd-kube-api-access-gg59n\") pod \"test-curl-7853-139915260888816\" (UID: \"54aecf37-b3e7-40f6-9a18-bc0c45d471bd\") " pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:01.440334 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.440309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:01.567256 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.567228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:17:01.570041 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:17:01.570018 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54aecf37_b3e7_40f6_9a18_bc0c45d471bd.slice/crio-d92a0f3b4a62700e4067d0a423e4435bdfcb8e69ded51a7037f2383610f2daa6 WatchSource:0}: Error finding container d92a0f3b4a62700e4067d0a423e4435bdfcb8e69ded51a7037f2383610f2daa6: Status 404 returned error can't find the container with id d92a0f3b4a62700e4067d0a423e4435bdfcb8e69ded51a7037f2383610f2daa6 Jul 15 14:17:01.831793 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.831708 2576 scope.go:117] "RemoveContainer" containerID="2f5cd07cf7501b26d1562d9f23232774de79b5f115bd3d7856f0c9f0c044d77c" Jul 15 14:17:01.831957 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.831716 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:01.833000 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:01.832951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915260888816" event={"ID":"54aecf37-b3e7-40f6-9a18-bc0c45d471bd","Type":"ContainerStarted","Data":"d92a0f3b4a62700e4067d0a423e4435bdfcb8e69ded51a7037f2383610f2daa6"} Jul 15 14:17:02.840493 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:02.840458 2576 generic.go:356] "Generic (PLEG): container finished" podID="54aecf37-b3e7-40f6-9a18-bc0c45d471bd" containerID="617d8c4d983c610026b2aa2601b239269f3581a83392e3f17dae0591784f1a39" exitCode=0 Jul 15 14:17:02.840493 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:02.840496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915260888816" event={"ID":"54aecf37-b3e7-40f6-9a18-bc0c45d471bd","Type":"ContainerDied","Data":"617d8c4d983c610026b2aa2601b239269f3581a83392e3f17dae0591784f1a39"} Jul 15 14:17:03.096682 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:03.096654 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e2c5726-360b-4f8d-acb4-02a44475c077" path="/var/lib/kubelet/pods/0e2c5726-360b-4f8d-acb4-02a44475c077/volumes" Jul 15 14:17:03.484644 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:03.484547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:17:03.973086 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:03.973060 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:03.994307 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:03.994275 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7853-139915260888816_54aecf37-b3e7-40f6-9a18-bc0c45d471bd/test-curl-7853-139915260888816/0.log" Jul 15 14:17:04.020800 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:04.020765 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:17:04.024493 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:04.024469 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-139915260888816"] Jul 15 14:17:04.025839 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:04.025821 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg59n\" (UniqueName: \"kubernetes.io/projected/54aecf37-b3e7-40f6-9a18-bc0c45d471bd-kube-api-access-gg59n\") pod \"54aecf37-b3e7-40f6-9a18-bc0c45d471bd\" (UID: \"54aecf37-b3e7-40f6-9a18-bc0c45d471bd\") " Jul 15 14:17:04.027797 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:04.027770 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54aecf37-b3e7-40f6-9a18-bc0c45d471bd-kube-api-access-gg59n" (OuterVolumeSpecName: "kube-api-access-gg59n") pod "54aecf37-b3e7-40f6-9a18-bc0c45d471bd" (UID: "54aecf37-b3e7-40f6-9a18-bc0c45d471bd"). InnerVolumeSpecName "kube-api-access-gg59n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:17:04.126662 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:04.126625 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg59n\" (UniqueName: \"kubernetes.io/projected/54aecf37-b3e7-40f6-9a18-bc0c45d471bd-kube-api-access-gg59n\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:04.850983 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:04.850955 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92a0f3b4a62700e4067d0a423e4435bdfcb8e69ded51a7037f2383610f2daa6" Jul 15 14:17:04.850983 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:04.850978 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:04.860370 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:17:04.860340 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7853-139915260888816\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="54aecf37-b3e7-40f6-9a18-bc0c45d471bd" pod="opendatahub/test-curl-7853-139915260888816" Jul 15 14:17:05.096775 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:05.096740 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54aecf37-b3e7-40f6-9a18-bc0c45d471bd" path="/var/lib/kubelet/pods/54aecf37-b3e7-40f6-9a18-bc0c45d471bd/volumes" Jul 15 14:17:05.885632 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:05.885590 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" podUID="c1b1e138-712f-4f55-8fe3-289411b1235c" containerName="main" containerID="cri-o://b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f" gracePeriod=45 Jul 15 14:17:06.135958 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.135899 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:17:06.146421 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146386 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwtzp\" (UniqueName: \"kubernetes.io/projected/c1b1e138-712f-4f55-8fe3-289411b1235c-kube-api-access-rwtzp\") pod \"c1b1e138-712f-4f55-8fe3-289411b1235c\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " Jul 15 14:17:06.146544 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146527 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-home\") pod \"c1b1e138-712f-4f55-8fe3-289411b1235c\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " Jul 15 14:17:06.146579 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146569 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-dshm\") pod \"c1b1e138-712f-4f55-8fe3-289411b1235c\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " Jul 15 14:17:06.146623 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146599 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-tmp-dir\") pod \"c1b1e138-712f-4f55-8fe3-289411b1235c\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " Jul 15 14:17:06.146654 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b1e138-712f-4f55-8fe3-289411b1235c-tls-certs\") pod \"c1b1e138-712f-4f55-8fe3-289411b1235c\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " Jul 15 14:17:06.146687 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146665 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-model-cache\") pod \"c1b1e138-712f-4f55-8fe3-289411b1235c\" (UID: \"c1b1e138-712f-4f55-8fe3-289411b1235c\") " Jul 15 14:17:06.147133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146831 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-home" (OuterVolumeSpecName: "home") pod "c1b1e138-712f-4f55-8fe3-289411b1235c" (UID: "c1b1e138-712f-4f55-8fe3-289411b1235c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:06.147133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146883 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1b1e138-712f-4f55-8fe3-289411b1235c" (UID: "c1b1e138-712f-4f55-8fe3-289411b1235c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:06.147133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.146989 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:06.147133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.147007 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:06.147133 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.147092 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1b1e138-712f-4f55-8fe3-289411b1235c" (UID: "c1b1e138-712f-4f55-8fe3-289411b1235c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:06.148677 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.148656 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-dshm" (OuterVolumeSpecName: "dshm") pod "c1b1e138-712f-4f55-8fe3-289411b1235c" (UID: "c1b1e138-712f-4f55-8fe3-289411b1235c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:06.148778 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.148691 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b1e138-712f-4f55-8fe3-289411b1235c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1b1e138-712f-4f55-8fe3-289411b1235c" (UID: "c1b1e138-712f-4f55-8fe3-289411b1235c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:17:06.148778 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.148707 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b1e138-712f-4f55-8fe3-289411b1235c-kube-api-access-rwtzp" (OuterVolumeSpecName: "kube-api-access-rwtzp") pod "c1b1e138-712f-4f55-8fe3-289411b1235c" (UID: "c1b1e138-712f-4f55-8fe3-289411b1235c"). InnerVolumeSpecName "kube-api-access-rwtzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:17:06.248167 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.248125 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:06.248167 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.248154 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b1e138-712f-4f55-8fe3-289411b1235c-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:06.248167 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.248164 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b1e138-712f-4f55-8fe3-289411b1235c-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:06.248167 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.248172 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtzp\" (UniqueName: \"kubernetes.io/projected/c1b1e138-712f-4f55-8fe3-289411b1235c-kube-api-access-rwtzp\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:06.694935 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.694902 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7853-139915262174528"] Jul 15 14:17:06.695282 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.695270 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b1e138-712f-4f55-8fe3-289411b1235c" containerName="main" Jul 15 14:17:06.695323 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.695283 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b1e138-712f-4f55-8fe3-289411b1235c" containerName="main" Jul 15 14:17:06.695323 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.695299 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54aecf37-b3e7-40f6-9a18-bc0c45d471bd" containerName="test-curl-7853-139915260888816" Jul 15 14:17:06.695323 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.695305 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="54aecf37-b3e7-40f6-9a18-bc0c45d471bd" containerName="test-curl-7853-139915260888816" Jul 15 14:17:06.695429 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.695370 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b1e138-712f-4f55-8fe3-289411b1235c" containerName="main" Jul 15 14:17:06.695429 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.695381 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="54aecf37-b3e7-40f6-9a18-bc0c45d471bd" containerName="test-curl-7853-139915260888816" Jul 15 14:17:06.698502 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.698484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915262174528" Jul 15 14:17:06.700671 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.700650 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wljdh\"" Jul 15 14:17:06.710036 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.710015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915262174528"] Jul 15 14:17:06.752156 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.752127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7st\" (UniqueName: \"kubernetes.io/projected/e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053-kube-api-access-8r7st\") pod \"test-curl-7853-139915262174528\" (UID: \"e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053\") " pod="opendatahub/test-curl-7853-139915262174528" Jul 15 14:17:06.853236 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.853203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7st\" (UniqueName: \"kubernetes.io/projected/e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053-kube-api-access-8r7st\") pod \"test-curl-7853-139915262174528\" (UID: \"e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053\") " pod="opendatahub/test-curl-7853-139915262174528" Jul 15 14:17:06.861736 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.861710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7st\" (UniqueName: \"kubernetes.io/projected/e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053-kube-api-access-8r7st\") pod \"test-curl-7853-139915262174528\" (UID: \"e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053\") " pod="opendatahub/test-curl-7853-139915262174528" Jul 15 14:17:06.862053 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.862029 2576 generic.go:356] "Generic (PLEG): container finished" podID="c1b1e138-712f-4f55-8fe3-289411b1235c" containerID="b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f" exitCode=0 Jul 15 14:17:06.862154 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.862130 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" Jul 15 14:17:06.862154 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.862136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" event={"ID":"c1b1e138-712f-4f55-8fe3-289411b1235c","Type":"ContainerDied","Data":"b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f"} Jul 15 14:17:06.862263 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.862177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt" event={"ID":"c1b1e138-712f-4f55-8fe3-289411b1235c","Type":"ContainerDied","Data":"904123df4671e94e10cefbd2a84ab10c771c4e0bed74682658b23874e85ba541"} Jul 15 14:17:06.862263 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.862202 2576 scope.go:117] "RemoveContainer" containerID="b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f" Jul 15 14:17:06.871148 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.871132 2576 scope.go:117] "RemoveContainer" containerID="b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f" Jul 15 14:17:06.871389 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:17:06.871372 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f\": container with ID starting with b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f not found: ID does not exist" containerID="b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f" Jul 15 14:17:06.871463 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.871413 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f"} err="failed to get container status \"b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f\": rpc error: code = NotFound desc = could not find container \"b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f\": container with ID starting with b3ca6458217b2842eb14b4ca4506d2276b83c3fcff3fbb801fe89f854382fd0f not found: ID does not exist" Jul 15 14:17:06.890887 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.890865 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt"] Jul 15 14:17:06.900490 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:06.900468 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bfdd40/rate-test-model-bfdd40-kserve-6d4576fcb7-xr9kt"] Jul 15 14:17:07.007607 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:07.007520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915262174528" Jul 15 14:17:07.097246 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:07.097213 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b1e138-712f-4f55-8fe3-289411b1235c" path="/var/lib/kubelet/pods/c1b1e138-712f-4f55-8fe3-289411b1235c/volumes" Jul 15 14:17:07.154877 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:07.154821 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7853-139915262174528"] Jul 15 14:17:07.157761 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:17:07.157725 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8cc44bc_0b2e_4a73_ac9c_3b7ecc030053.slice/crio-bb53448d177c1a00eee2ac760f611680379aa9b54838bf8185b5d0ec48435aab WatchSource:0}: Error finding container bb53448d177c1a00eee2ac760f611680379aa9b54838bf8185b5d0ec48435aab: Status 404 returned error can't find the container with id bb53448d177c1a00eee2ac760f611680379aa9b54838bf8185b5d0ec48435aab Jul 15 14:17:07.868715 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:07.868677 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053" containerID="57dbe1c697f552b93b0aba55727b797f09714f06549885a51f5ab81ede25a9c0" exitCode=0 Jul 15 14:17:07.868892 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:07.868732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915262174528" event={"ID":"e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053","Type":"ContainerDied","Data":"57dbe1c697f552b93b0aba55727b797f09714f06549885a51f5ab81ede25a9c0"} Jul 15 14:17:07.868892 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:07.868775 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7853-139915262174528" event={"ID":"e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053","Type":"ContainerStarted","Data":"bb53448d177c1a00eee2ac760f611680379aa9b54838bf8185b5d0ec48435aab"} Jul 15 14:17:09.002640 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.002611 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915262174528" Jul 15 14:17:09.027212 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.027191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7853-139915262174528_e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053/test-curl-7853-139915262174528/0.log" Jul 15 14:17:09.052970 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.052944 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7853-139915262174528"] Jul 15 14:17:09.060649 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.060626 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7853-139915262174528"] Jul 15 14:17:09.073283 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.073263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r7st\" (UniqueName: \"kubernetes.io/projected/e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053-kube-api-access-8r7st\") pod \"e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053\" (UID: \"e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053\") " Jul 15 14:17:09.075274 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.075253 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053-kube-api-access-8r7st" (OuterVolumeSpecName: "kube-api-access-8r7st") pod "e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053" (UID: "e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053"). InnerVolumeSpecName "kube-api-access-8r7st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:17:09.097213 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.097189 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053" path="/var/lib/kubelet/pods/e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053/volumes" Jul 15 14:17:09.175037 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.174976 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7st\" (UniqueName: \"kubernetes.io/projected/e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053-kube-api-access-8r7st\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:09.881596 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.881567 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7853-139915262174528" Jul 15 14:17:09.881802 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:09.881622 2576 scope.go:117] "RemoveContainer" containerID="57dbe1c697f552b93b0aba55727b797f09714f06549885a51f5ab81ede25a9c0" Jul 15 14:17:23.298859 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:23.298824 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:17:35.117250 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:35.117215 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd"] Jul 15 14:17:35.117768 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:35.117457 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" containerID="cri-o://b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc" gracePeriod=30 Jul 15 14:17:40.364333 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.364306 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:17:40.465689 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465596 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kll7x\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-kube-api-access-kll7x\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465689 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-istio-token\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465700 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-credential-socket\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cb802c-3122-4e4d-8504-0394abd7877a-istiod-ca-cert\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-certs\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465857 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cb802c-3122-4e4d-8504-0394abd7877a-istio-podinfo\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465905 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-data\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-envoy\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.465994 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.465978 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-socket\") pod \"87cb802c-3122-4e4d-8504-0394abd7877a\" (UID: \"87cb802c-3122-4e4d-8504-0394abd7877a\") " Jul 15 14:17:40.466339 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466028 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:40.466339 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466081 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:40.466339 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466127 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cb802c-3122-4e4d-8504-0394abd7877a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:17:40.466339 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:40.466578 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466473 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.466578 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466490 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.466578 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466505 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87cb802c-3122-4e4d-8504-0394abd7877a-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.466578 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466518 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.466983 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.466957 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-data" (OuterVolumeSpecName: "istio-data") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:40.468109 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.468082 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-istio-token" (OuterVolumeSpecName: "istio-token") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:17:40.468231 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.468132 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87cb802c-3122-4e4d-8504-0394abd7877a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 14:17:40.468317 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.468299 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-kube-api-access-kll7x" (OuterVolumeSpecName: "kube-api-access-kll7x") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "kube-api-access-kll7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:17:40.468369 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.468337 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87cb802c-3122-4e4d-8504-0394abd7877a" (UID: "87cb802c-3122-4e4d-8504-0394abd7877a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:17:40.567005 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.566971 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.567005 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.567000 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87cb802c-3122-4e4d-8504-0394abd7877a-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.567005 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.567010 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.567238 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.567018 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87cb802c-3122-4e4d-8504-0394abd7877a-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:40.567238 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:40.567027 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kll7x\" (UniqueName: \"kubernetes.io/projected/87cb802c-3122-4e4d-8504-0394abd7877a-kube-api-access-kll7x\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\"" Jul 15 14:17:41.038343 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.038305 2576 generic.go:356] "Generic (PLEG): container finished" podID="87cb802c-3122-4e4d-8504-0394abd7877a" containerID="b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc" exitCode=0 Jul 15 14:17:41.038540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.038358 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" Jul 15 14:17:41.038540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.038419 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" event={"ID":"87cb802c-3122-4e4d-8504-0394abd7877a","Type":"ContainerDied","Data":"b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc"} Jul 15 14:17:41.038540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.038476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd" event={"ID":"87cb802c-3122-4e4d-8504-0394abd7877a","Type":"ContainerDied","Data":"11fce052ec3570fb9b0c25f6f07229621d0b3467758321504a2e6477a932caaa"} Jul 15 14:17:41.038540 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.038501 2576 scope.go:117] "RemoveContainer" containerID="b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc" Jul 15 14:17:41.047878 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.047863 2576 scope.go:117] "RemoveContainer" containerID="89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655" Jul 15 14:17:41.055421 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.055391 2576 scope.go:117] "RemoveContainer" containerID="b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc" Jul 15 14:17:41.055660 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:17:41.055639 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc\": container with ID starting with b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc not found: ID does not exist" containerID="b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc" Jul 15 14:17:41.055726 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.055667 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc"} err="failed to get container status \"b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc\": rpc error: code = NotFound desc = could not find container \"b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc\": container with ID starting with b3537064fe46ce16d199200344f92bfc8bbb7b13db86613606a981976edf2dbc not found: ID does not exist" Jul 15 14:17:41.055726 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.055685 2576 scope.go:117] "RemoveContainer" containerID="89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655" Jul 15 14:17:41.055963 ip-10-0-129-47 kubenswrapper[2576]: E0715 14:17:41.055944 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655\": container with ID starting with 89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655 not found: ID does not exist" containerID="89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655" Jul 15 14:17:41.056014 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.055969 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655"} err="failed to get container status \"89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655\": rpc error: code = NotFound desc = could not find container \"89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655\": container with ID starting with 89b785c11842b620423c5e4fa2e7eb741efea9e351c82c070aedabec457f6655 not found: ID does not exist" Jul 15 14:17:41.063905 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.063885 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd"] Jul 15 14:17:41.068265 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.068246 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-2723e9-openshift-default-764b5cb8ff-bk8qd"] Jul 15 14:17:41.096046 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:41.096021 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" path="/var/lib/kubelet/pods/87cb802c-3122-4e4d-8504-0394abd7877a/volumes" Jul 15 14:17:49.385894 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:17:49.385861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-g8228"] Jul 15 14:18:08.869070 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:08.869039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84978f4d9d-5px9w_7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1/copy-manifests/0.log" Jul 15 14:18:08.876304 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:08.876282 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84978f4d9d-5px9w_7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1/manager/0.log" Jul 15 14:18:13.847940 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:13.847877 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g8228_d56ad4c7-45a8-4626-9100-afec43bb2d38/limitador/0.log" Jul 15 14:18:14.646573 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:14.646539 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9ddcd897f-gkpkj_cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4/kube-auth-proxy/0.log" Jul 15 14:18:14.902734 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:14.902661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nv52m_105d60bd-9431-4f83-98a2-1948172c7c5c/istio-proxy/1.log" Jul 15 14:18:16.151221 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:16.151187 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db4x949_448c25a8-9d10-44d8-87bf-c0b22818cc28/main/0.log" Jul 15 14:18:16.413542 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:16.413429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-wlt85_ca01a4d6-2232-4036-a01d-3cd50eb1523b/main/0.log" Jul 15 14:18:22.779188 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779157 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-858vm/must-gather-qml6p"] Jul 15 14:18:22.779635 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779532 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" Jul 15 14:18:22.779635 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779545 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" Jul 15 14:18:22.779635 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779556 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" Jul 15 14:18:22.779635 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779562 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" Jul 15 14:18:22.779635 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779584 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053" containerName="test-curl-7853-139915262174528" Jul 15 14:18:22.779635 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779591 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053" containerName="test-curl-7853-139915262174528" Jul 15 14:18:22.779836 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779650 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" Jul 15 14:18:22.779836 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779663 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8cc44bc-0b2e-4a73-ac9c-3b7ecc030053" containerName="test-curl-7853-139915262174528" Jul 15 14:18:22.779836 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.779771 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="87cb802c-3122-4e4d-8504-0394abd7877a" containerName="istio-proxy" Jul 15 14:18:22.784123 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.784099 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:22.788011 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.787987 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-858vm\"/\"kube-root-ca.crt\"" Jul 15 14:18:22.788588 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.788568 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-858vm\"/\"openshift-service-ca.crt\"" Jul 15 14:18:22.788689 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.788621 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-858vm\"/\"default-dockercfg-cx228\"" Jul 15 14:18:22.794825 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.794804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-858vm/must-gather-qml6p"] Jul 15 14:18:22.872145 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.872100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ntll\" (UniqueName: \"kubernetes.io/projected/d18b5614-0543-4157-a9b5-04ab1c06bc36-kube-api-access-8ntll\") pod \"must-gather-qml6p\" (UID: \"d18b5614-0543-4157-a9b5-04ab1c06bc36\") " pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:22.872341 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.872198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d18b5614-0543-4157-a9b5-04ab1c06bc36-must-gather-output\") pod \"must-gather-qml6p\" (UID: \"d18b5614-0543-4157-a9b5-04ab1c06bc36\") " pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:22.972810 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.972774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ntll\" (UniqueName: \"kubernetes.io/projected/d18b5614-0543-4157-a9b5-04ab1c06bc36-kube-api-access-8ntll\") pod \"must-gather-qml6p\" (UID: \"d18b5614-0543-4157-a9b5-04ab1c06bc36\") " pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:22.973059 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.972859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d18b5614-0543-4157-a9b5-04ab1c06bc36-must-gather-output\") pod \"must-gather-qml6p\" (UID: \"d18b5614-0543-4157-a9b5-04ab1c06bc36\") " pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:22.973257 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.973234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d18b5614-0543-4157-a9b5-04ab1c06bc36-must-gather-output\") pod \"must-gather-qml6p\" (UID: \"d18b5614-0543-4157-a9b5-04ab1c06bc36\") " pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:22.981196 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:22.981172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ntll\" (UniqueName: \"kubernetes.io/projected/d18b5614-0543-4157-a9b5-04ab1c06bc36-kube-api-access-8ntll\") pod \"must-gather-qml6p\" (UID: \"d18b5614-0543-4157-a9b5-04ab1c06bc36\") " pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:23.093180 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:23.093093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-858vm/must-gather-qml6p" Jul 15 14:18:23.257576 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:23.257539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-858vm/must-gather-qml6p"] Jul 15 14:18:23.260345 ip-10-0-129-47 kubenswrapper[2576]: W0715 14:18:23.260315 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd18b5614_0543_4157_a9b5_04ab1c06bc36.slice/crio-cc4949c6a910c75715a36aa0472b10a11062b7dee3c483c3dfc79707fd5f223a WatchSource:0}: Error finding container cc4949c6a910c75715a36aa0472b10a11062b7dee3c483c3dfc79707fd5f223a: Status 404 returned error can't find the container with id cc4949c6a910c75715a36aa0472b10a11062b7dee3c483c3dfc79707fd5f223a Jul 15 14:18:24.246118 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:24.246080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-858vm/must-gather-qml6p" event={"ID":"d18b5614-0543-4157-a9b5-04ab1c06bc36","Type":"ContainerStarted","Data":"880d1f84a14fee41e05cfafc33690e2272c7e75cbfcbd8b5be910cc5bd6a1af3"} Jul 15 14:18:24.246514 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:24.246129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-858vm/must-gather-qml6p" event={"ID":"d18b5614-0543-4157-a9b5-04ab1c06bc36","Type":"ContainerStarted","Data":"cc4949c6a910c75715a36aa0472b10a11062b7dee3c483c3dfc79707fd5f223a"} Jul 15 14:18:25.254830 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:25.254782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-858vm/must-gather-qml6p" event={"ID":"d18b5614-0543-4157-a9b5-04ab1c06bc36","Type":"ContainerStarted","Data":"a17277c7f928cfb3544810852bed5e619cf4a8fbd30e01c0279b4f6bbd06a1ca"} Jul 15 14:18:25.273369 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:25.273270 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-858vm/must-gather-qml6p" podStartSLOduration=2.483650843 podStartE2EDuration="3.273246616s" totalImagesPullingTime="789.595773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:18:25.271769497 +0000 UTC m=+2946.789393724" watchObservedRunningTime="2026-07-15 14:18:25.273246616 +0000 UTC m=+2946.790870816" Jul 15 14:18:25.759010 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:25.758974 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tkwn8_eafabcae-943c-4420-b244-a96aa27bedb9/global-pull-secret-syncer/0.log" Jul 15 14:18:25.918369 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:25.918335 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q657c_53adf7c9-4b8b-4f34-aba1-fffe96e1b434/konnectivity-agent/0.log" Jul 15 14:18:25.966142 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:25.966108 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-47.ec2.internal_f903a0bdcc0b6d32ccdde50ff8a5e596/haproxy/0.log" Jul 15 14:18:32.171969 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:32.171929 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-g8228_d56ad4c7-45a8-4626-9100-afec43bb2d38/limitador/0.log" Jul 15 14:18:33.976427 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:33.976384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55566a89-2540-4586-895e-758962b2fec6/alertmanager/0.log" Jul 15 14:18:34.002859 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.002793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55566a89-2540-4586-895e-758962b2fec6/config-reloader/0.log" Jul 15 14:18:34.026848 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.026817 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55566a89-2540-4586-895e-758962b2fec6/kube-rbac-proxy-web/0.log" Jul 15 14:18:34.051816 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.051778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55566a89-2540-4586-895e-758962b2fec6/kube-rbac-proxy/0.log" Jul 15 14:18:34.076178 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.076148 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55566a89-2540-4586-895e-758962b2fec6/kube-rbac-proxy-metric/0.log" Jul 15 14:18:34.103765 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.103728 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55566a89-2540-4586-895e-758962b2fec6/prom-label-proxy/0.log" Jul 15 14:18:34.128925 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.128896 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_55566a89-2540-4586-895e-758962b2fec6/init-config-reloader/0.log" Jul 15 14:18:34.203123 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.203075 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-kg5vz_41cd44c2-5d29-46c9-ae23-b1efb285f8c7/kube-state-metrics/0.log" Jul 15 14:18:34.231024 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.230946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-kg5vz_41cd44c2-5d29-46c9-ae23-b1efb285f8c7/kube-rbac-proxy-main/0.log" Jul 15 14:18:34.260873 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.260837 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-kg5vz_41cd44c2-5d29-46c9-ae23-b1efb285f8c7/kube-rbac-proxy-self/0.log" Jul 15 14:18:34.289876 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.289848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bd7dd65bc-swdqf_d64398d2-26df-4757-a2ac-65cb057b74e4/metrics-server/0.log" Jul 15 14:18:34.320365 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.320325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-zjsgb_182235c3-4fba-4aac-b1a8-33834d39d0c9/monitoring-plugin/0.log" Jul 15 14:18:34.533179 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.533101 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2d5s_3299a86f-8ce9-442e-8c4d-1b40cf4e49a6/node-exporter/0.log" Jul 15 14:18:34.559830 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.559803 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2d5s_3299a86f-8ce9-442e-8c4d-1b40cf4e49a6/kube-rbac-proxy/0.log" Jul 15 14:18:34.584789 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.584755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2d5s_3299a86f-8ce9-442e-8c4d-1b40cf4e49a6/init-textfile/0.log" Jul 15 14:18:34.701500 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.701460 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-z6mn6_eda49ddc-ac2f-41ce-a01e-5c0efddf15f8/kube-rbac-proxy-main/0.log" Jul 15 14:18:34.727547 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.727513 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-z6mn6_eda49ddc-ac2f-41ce-a01e-5c0efddf15f8/kube-rbac-proxy-self/0.log" Jul 15 14:18:34.753825 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.753795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-z6mn6_eda49ddc-ac2f-41ce-a01e-5c0efddf15f8/openshift-state-metrics/0.log" Jul 15 14:18:34.787681 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.787593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3eb5954e-f956-444a-9960-9db974faa315/prometheus/0.log" Jul 15 14:18:34.820641 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.820615 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3eb5954e-f956-444a-9960-9db974faa315/config-reloader/0.log" Jul 15 14:18:34.844581 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.844539 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3eb5954e-f956-444a-9960-9db974faa315/thanos-sidecar/0.log" Jul 15 14:18:34.872204 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.872173 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3eb5954e-f956-444a-9960-9db974faa315/kube-rbac-proxy-web/0.log" Jul 15 14:18:34.902733 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.902707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3eb5954e-f956-444a-9960-9db974faa315/kube-rbac-proxy/0.log" Jul 15 14:18:34.929640 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.929613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3eb5954e-f956-444a-9960-9db974faa315/kube-rbac-proxy-thanos/0.log" Jul 15 14:18:34.968036 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.967994 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3eb5954e-f956-444a-9960-9db974faa315/init-config-reloader/0.log" Jul 15 14:18:34.996077 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:34.996044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-cjp8v_2623a670-f328-48f5-9e06-d77bb36bc903/prometheus-operator/0.log" Jul 15 14:18:35.020033 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.019995 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-cjp8v_2623a670-f328-48f5-9e06-d77bb36bc903/kube-rbac-proxy/0.log" Jul 15 14:18:35.046107 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.046026 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-bm9dd_40166e11-b01b-449c-9126-30bac206854e/prometheus-operator-admission-webhook/0.log" Jul 15 14:18:35.074960 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.074919 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9c794874-wmm96_6fc6e973-642a-4001-9011-38705d5dbce0/telemeter-client/0.log" Jul 15 14:18:35.105766 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.105716 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9c794874-wmm96_6fc6e973-642a-4001-9011-38705d5dbce0/reload/0.log" Jul 15 14:18:35.133745 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.133709 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b9c794874-wmm96_6fc6e973-642a-4001-9011-38705d5dbce0/kube-rbac-proxy/0.log" Jul 15 14:18:35.173933 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.173903 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc85f78c5-4wlq6_5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8/thanos-query/0.log" Jul 15 14:18:35.196229 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.196201 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc85f78c5-4wlq6_5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8/kube-rbac-proxy-web/0.log" Jul 15 14:18:35.220111 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.220070 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc85f78c5-4wlq6_5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8/kube-rbac-proxy/0.log" Jul 15 14:18:35.243333 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.243303 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc85f78c5-4wlq6_5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8/prom-label-proxy/0.log" Jul 15 14:18:35.278551 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.278526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc85f78c5-4wlq6_5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8/kube-rbac-proxy-rules/0.log" Jul 15 14:18:35.301899 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.301817 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fc85f78c5-4wlq6_5ea9df3b-bd08-4f7b-a082-bd13ff45f2f8/kube-rbac-proxy-metrics/0.log" Jul 15 14:18:35.789212 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.789155 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd"] Jul 15 14:18:35.801469 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.801434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:35.803072 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.803033 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd"] Jul 15 14:18:35.913989 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.913942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-proc\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:35.914191 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.914026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-lib-modules\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:35.914191 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.914159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdpv\" (UniqueName: \"kubernetes.io/projected/5b1a45a1-3ba2-442b-95e6-61055b566c01-kube-api-access-4tdpv\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:35.914315 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.914205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-sys\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:35.914315 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:35.914249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-podres\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.015648 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.015596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdpv\" (UniqueName: \"kubernetes.io/projected/5b1a45a1-3ba2-442b-95e6-61055b566c01-kube-api-access-4tdpv\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016194 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.015674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-sys\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016194 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.015719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-podres\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016194 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.015753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-proc\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016194 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.015801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-lib-modules\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016194 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.015993 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-lib-modules\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016521 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.016454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-sys\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016583 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.016564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-podres\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.016640 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.016613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b1a45a1-3ba2-442b-95e6-61055b566c01-proc\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.024875 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.024847 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdpv\" (UniqueName: \"kubernetes.io/projected/5b1a45a1-3ba2-442b-95e6-61055b566c01-kube-api-access-4tdpv\") pod \"perf-node-gather-daemonset-vrgfd\" (UID: \"5b1a45a1-3ba2-442b-95e6-61055b566c01\") " pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.123047 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.123010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:36.299051 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.299011 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd"] Jul 15 14:18:36.342795 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:36.342760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" event={"ID":"5b1a45a1-3ba2-442b-95e6-61055b566c01","Type":"ContainerStarted","Data":"66c1a73ab58bdc16ac3548e2b96279415f1287736891eed015ac4900cc6b96b0"} Jul 15 14:18:37.351519 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:37.351483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" event={"ID":"5b1a45a1-3ba2-442b-95e6-61055b566c01","Type":"ContainerStarted","Data":"f21fabbc6f829a395ebc5778662c9e2615fce0eeb8a402be4e68184ee9ab8b4f"} Jul 15 14:18:37.351942 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:37.351607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:37.387984 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:37.387906 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" podStartSLOduration=2.387883128 podStartE2EDuration="2.387883128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:18:37.387601787 +0000 UTC m=+2958.905225988" watchObservedRunningTime="2026-07-15 14:18:37.387883128 +0000 UTC m=+2958.905507337" Jul 15 14:18:39.483031 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:39.483001 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dgm75_8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6/dns/0.log" Jul 15 14:18:39.511106 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:39.511078 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dgm75_8eb3cce7-d6f7-4803-99fc-9fa7ff79a8f6/kube-rbac-proxy/0.log" Jul 15 14:18:39.707972 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:39.707946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spfvj_77907d2e-8cca-44e7-a03b-8b34bf3bd2bf/dns-node-resolver/0.log" Jul 15 14:18:40.343672 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:40.343638 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkm5q_d29ac1b1-707e-4791-97f3-62acd643c83d/node-ca/0.log" Jul 15 14:18:41.630003 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:41.629926 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9ddcd897f-gkpkj_cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4/kube-auth-proxy/0.log" Jul 15 14:18:41.713685 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:41.713655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nv52m_105d60bd-9431-4f83-98a2-1948172c7c5c/istio-proxy/0.log" Jul 15 14:18:41.749178 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:41.749145 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nv52m_105d60bd-9431-4f83-98a2-1948172c7c5c/istio-proxy/1.log" Jul 15 14:18:42.670630 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:42.670598 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5gfdp_eb60e8c3-59de-4093-9c04-d6904b54625f/serve-healthcheck-canary/0.log" Jul 15 14:18:43.367556 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:43.367528 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-858vm/perf-node-gather-daemonset-vrgfd" Jul 15 14:18:43.453934 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:43.453897 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z784n_27ca2901-5e9e-42a7-ad5f-1b586f3667bf/kube-rbac-proxy/0.log" Jul 15 14:18:43.479469 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:43.479441 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z784n_27ca2901-5e9e-42a7-ad5f-1b586f3667bf/exporter/0.log" Jul 15 14:18:43.504596 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:43.504529 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z784n_27ca2901-5e9e-42a7-ad5f-1b586f3667bf/extractor/0.log" Jul 15 14:18:46.462769 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:46.462730 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84978f4d9d-5px9w_7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1/manager/0.log" Jul 15 14:18:46.493880 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:46.493850 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84978f4d9d-5px9w_7d2c3e2e-6eb1-472d-8a2e-cf416b2ec7e1/copy-manifests/0.log" Jul 15 14:18:46.589417 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:46.589382 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9ddcd897f-gkpkj_cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4/kube-auth-proxy/0.log" Jul 15 14:18:46.633732 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:46.633689 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nv52m_105d60bd-9431-4f83-98a2-1948172c7c5c/istio-proxy/0.log" Jul 15 14:18:46.672743 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:46.672707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nv52m_105d60bd-9431-4f83-98a2-1948172c7c5c/istio-proxy/1.log" Jul 15 14:18:48.074236 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:48.074201 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74d75685f8-vl2jl_c87885ef-a3e9-41f1-b969-3bf7168fb7fb/manager/0.log" Jul 15 14:18:48.313108 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:48.313032 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9ddcd897f-gkpkj_cfbb911c-dd9d-4b00-bc6d-d2afbd3a1df4/kube-auth-proxy/0.log" Jul 15 14:18:48.362731 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:48.362698 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nv52m_105d60bd-9431-4f83-98a2-1948172c7c5c/istio-proxy/0.log" Jul 15 14:18:48.398365 ip-10-0-129-47 kubenswrapper[2576]: I0715 14:18:48.398333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nv52m_105d60bd-9431-4f83-98a2-1948172c7c5c/istio-proxy/1.log"