Jul 30 20:21:13.646107 ip-10-0-131-223 systemd[1]: Starting Kubernetes Kubelet... Jul 30 20:21:13.949480 ip-10-0-131-223 kubenswrapper[2537]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 20:21:13.949480 ip-10-0-131-223 kubenswrapper[2537]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 20:21:13.949480 ip-10-0-131-223 kubenswrapper[2537]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 20:21:13.949480 ip-10-0-131-223 kubenswrapper[2537]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 20:21:13.949480 ip-10-0-131-223 kubenswrapper[2537]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 20:21:13.950288 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.950118 2537 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 20:21:13.954334 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954314 2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:13.954334 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954333 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:13.954334 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954337 2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954343 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954348 2537 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954351 2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954355 2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954359 2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954363 2537 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954366 2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954372 2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954378 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954382 2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954387 2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954391 2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954395 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954400 2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954404 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954416 2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954420 2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954426 2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:13.954540 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954432 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954437 2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954442 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954446 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954450 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954454 2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954458 2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954463 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954467 2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954471 2537 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954476 2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954481 2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954485 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954490 2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954494 2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954500 2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954505 2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954509 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954513 2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954518 2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:13.955341 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954522 2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954526 2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954530 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954535 2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954539 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954544 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954548 2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954552 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954557 2537 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954561 2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954568 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954573 2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954578 2537 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954582 2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954586 2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954591 2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954595 2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954599 2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954603 2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954608 2537 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:13.955850 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954612 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954616 2537 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954621 2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954626 2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954630 2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954634 2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954638 2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954644 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954649 2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954653 2537 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954657 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954661 2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954666 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954670 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954674 2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954679 2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954683 2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954687 2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954691 2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954696 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:13.956383 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954701 2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954705 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954709 2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954715 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954719 2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954723 2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954730 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954734 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954741 2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954746 2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954751 2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954756 2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954761 2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954766 2537 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954771 2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954789 2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954793 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:13.956917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.954798 2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:13.958000 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.957983 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:13.958000 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.957997 2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:13.958000 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958002 2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:13.958000 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958006 2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958011 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958015 2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958019 2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958023 2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958026 2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958030 2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958035 2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958039 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958043 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958047 2537 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958050 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958054 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958058 2537 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958062 2537 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958067 2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958071 2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958075 2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958079 2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958082 2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:13.958230 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958086 2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958090 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958095 2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958099 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958103 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958107 2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958111 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958116 2537 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958120 2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958126 2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958133 2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958138 2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958142 2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958146 2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958152 2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958157 2537 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958161 2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958168 2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958174 2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:13.959085 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958178 2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958183 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958188 2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958192 2537 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958197 2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958201 2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958206 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958211 2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958215 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958220 2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958224 2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958228 2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958233 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958237 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958241 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958245 2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958251 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958256 2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958260 2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958264 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:13.959922 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958269 2537 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958273 2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958277 2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958281 2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958286 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958290 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958294 2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958300 2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958304 2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958309 2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958313 2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958317 2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958321 2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958326 2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958330 2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958334 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958338 2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958342 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958347 2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:13.960849 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958351 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958355 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958359 2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958364 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958368 2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958372 2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958376 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958381 2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958385 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958389 2537 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958393 2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958397 2537 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958402 2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958409 2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958414 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958419 2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958423 2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.958428 2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959064 2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:13.961467 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959074 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959079 2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959084 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959088 2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959092 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959096 2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959100 2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959104 2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959108 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959113 2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959118 2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959122 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959126 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959131 2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959135 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959139 2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959144 2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959149 2537 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959153 2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959157 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:13.962047 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959162 2537 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959169 2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959174 2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959179 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959184 2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959189 2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959193 2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959198 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959202 2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959206 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959210 2537 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959215 2537 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959219 2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959224 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959228 2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959232 2537 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959236 2537 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959240 2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959244 2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959248 2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:13.962721 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959253 2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959257 2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959261 2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959265 2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959269 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959273 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959277 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959282 2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959288 2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959294 2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959299 2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959304 2537 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959309 2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959313 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959318 2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959322 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959328 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959333 2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959337 2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:13.963289 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959341 2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959345 2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959349 2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959353 2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959357 2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959362 2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959366 2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959370 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959375 2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959380 2537 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959384 2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959388 2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959392 2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959396 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959402 2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959409 2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959413 2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959418 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959422 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959426 2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:13.963798 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959430 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959435 2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959439 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959443 2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959448 2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959453 2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959457 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959462 2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959466 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959471 2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959476 2537 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959480 2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959485 2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959489 2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959493 2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959498 2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959502 2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959506 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.959510 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959614 2537 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959625 2537 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 20:21:13.964346 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959636 2537 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959644 2537 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959651 2537 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959656 2537 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959663 2537 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959670 2537 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959676 2537 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959681 2537 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959686 2537 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959700 2537 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959706 2537 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959711 2537 flags.go:64] FLAG: --cgroup-root="" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959716 2537 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959721 2537 flags.go:64] FLAG: --client-ca-file="" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959725 2537 flags.go:64] FLAG: --cloud-provider="external" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959731 2537 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959737 2537 flags.go:64] FLAG: --cluster-domain="" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959742 2537 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959748 2537 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959753 2537 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959760 2537 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959766 2537 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959771 2537 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 20:21:13.964892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959791 2537 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959796 2537 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959802 2537 flags.go:64] FLAG: --contention-profiling="false" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959806 2537 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959811 2537 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959817 2537 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959822 2537 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959829 2537 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959834 2537 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959838 2537 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959843 2537 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959848 2537 flags.go:64] FLAG: --enable-server="true" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959853 2537 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959862 2537 flags.go:64] FLAG: --event-burst="100" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959867 2537 flags.go:64] FLAG: --event-qps="50" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959871 2537 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959876 2537 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959881 2537 flags.go:64] FLAG: --eviction-hard="" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959887 2537 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959892 2537 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959897 2537 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959902 2537 flags.go:64] FLAG: --eviction-soft="" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959907 2537 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959911 2537 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959922 2537 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 20:21:13.965442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959927 2537 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959931 2537 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959936 2537 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959941 2537 flags.go:64] FLAG: --feature-gates="" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959950 2537 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959955 2537 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959960 2537 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959965 2537 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959970 2537 flags.go:64] FLAG: --healthz-port="10248" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959976 2537 flags.go:64] FLAG: --help="false" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959981 2537 flags.go:64] FLAG: --hostname-override="ip-10-0-131-223.ec2.internal" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959986 2537 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959990 2537 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.959995 2537 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960001 2537 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960007 2537 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960011 2537 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960017 2537 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960022 2537 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960026 2537 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960031 2537 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960036 2537 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960041 2537 flags.go:64] FLAG: --kube-reserved="" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960047 2537 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 20:21:13.966122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960052 2537 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960057 2537 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960061 2537 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960066 2537 flags.go:64] FLAG: --lock-file="" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960071 2537 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960075 2537 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960080 2537 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960091 2537 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960095 2537 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960104 2537 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960109 2537 flags.go:64] FLAG: --logging-format="text" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960114 2537 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960122 2537 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960127 2537 flags.go:64] FLAG: --manifest-url="" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960131 2537 flags.go:64] FLAG: --manifest-url-header="" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960138 2537 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960143 2537 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960150 2537 flags.go:64] FLAG: --max-pods="110" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960155 2537 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960160 2537 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960164 2537 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960170 2537 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960174 2537 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960179 2537 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960184 2537 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 20:21:13.966749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960196 2537 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960201 2537 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960206 2537 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960211 2537 flags.go:64] FLAG: --pod-cidr="" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960216 2537 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960225 2537 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960230 2537 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960234 2537 flags.go:64] FLAG: --pods-per-core="0" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960239 2537 flags.go:64] FLAG: --port="10250" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960244 2537 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960248 2537 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f0cad6d6c10eb62a" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960254 2537 flags.go:64] FLAG: --qos-reserved="" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960259 2537 flags.go:64] FLAG: --read-only-port="10255" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960264 2537 flags.go:64] FLAG: --register-node="true" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960269 2537 flags.go:64] FLAG: --register-with-taints="" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960277 2537 flags.go:64] FLAG: --registry-burst="10" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960281 2537 flags.go:64] FLAG: --registry-qps="5" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960287 2537 flags.go:64] FLAG: --reserved-cpus="" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960291 2537 flags.go:64] FLAG: --reserved-memory="" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960300 2537 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960305 2537 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960310 2537 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960315 2537 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960319 2537 flags.go:64] FLAG: --runonce="false" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960324 2537 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 20:21:13.967410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960329 2537 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960333 2537 flags.go:64] FLAG: --seccomp-default="false" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960338 2537 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960343 2537 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960348 2537 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960353 2537 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960358 2537 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960363 2537 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960367 2537 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960372 2537 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960377 2537 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960382 2537 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960387 2537 flags.go:64] FLAG: --system-cgroups="" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960392 2537 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960401 2537 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960405 2537 flags.go:64] FLAG: --tls-cert-file="" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960410 2537 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960416 2537 flags.go:64] FLAG: --tls-min-version="" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960422 2537 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960426 2537 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960431 2537 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960436 2537 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960442 2537 flags.go:64] FLAG: --v="2" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960449 2537 flags.go:64] FLAG: --version="false" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960455 2537 flags.go:64] FLAG: --vmodule="" Jul 30 20:21:13.968040 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960462 2537 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.960470 2537 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960656 2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960663 2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960668 2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960674 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960679 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960683 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960688 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960692 2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960697 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960701 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960705 2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960709 2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960714 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960719 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960725 2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960731 2537 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960736 2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:13.968738 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960740 2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960745 2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960749 2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960753 2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960758 2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960762 2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960767 2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960771 2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960794 2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960799 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960806 2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960810 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960814 2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960818 2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960826 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960831 2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960836 2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960840 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960844 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960848 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:13.969313 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960852 2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960858 2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960864 2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960871 2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960877 2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960881 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960885 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960889 2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960894 2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960899 2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960903 2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960907 2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960911 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960916 2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960920 2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960924 2537 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960929 2537 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960933 2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960938 2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:13.969879 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960942 2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960947 2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960951 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960957 2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960961 2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960966 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960970 2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960976 2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960981 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960985 2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960990 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960994 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.960999 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961003 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961007 2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961011 2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961015 2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961020 2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961024 2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:13.970351 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961028 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961032 2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961037 2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961041 2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961045 2537 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961049 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961053 2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961057 2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961062 2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961066 2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961070 2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961075 2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961079 2537 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961083 2537 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961087 2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961091 2537 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961097 2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961101 2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961105 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961110 2537 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961116 2537 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:13.970873 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961120 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961125 2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.961129 2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.961656 2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.970409 2537 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.970428 2537 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.971038 2537 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.971046 2537 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971107 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971112 2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971116 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971119 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971122 2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971126 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971128 2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:13.971389 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971131 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971134 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971137 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971139 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971142 2537 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971145 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971147 2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971151 2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971154 2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971157 2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971160 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971162 2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971165 2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971168 2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971170 2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971173 2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971176 2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971179 2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971182 2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:13.971764 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971184 2537 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971187 2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971190 2537 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971193 2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971196 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971198 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971201 2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971204 2537 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971206 2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971209 2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971212 2537 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971214 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971217 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971219 2537 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971222 2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971225 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971227 2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971230 2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971233 2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:13.972242 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971235 2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971241 2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971246 2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971249 2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971252 2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971255 2537 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971259 2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971263 2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971266 2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971269 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971272 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971275 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971278 2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971280 2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971284 2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971286 2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971290 2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971293 2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971296 2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:13.972704 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971300 2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971303 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971307 2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971309 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971312 2537 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971315 2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971318 2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971321 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971324 2537 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971327 2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971329 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971332 2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971335 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971337 2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971340 2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971344 2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971347 2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971350 2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971352 2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971355 2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:13.973182 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971358 2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971361 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971363 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971366 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971369 2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971371 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971374 2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971377 2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971379 2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971382 2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971385 2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971388 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971390 2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971393 2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971396 2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:13.973664 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.971402 2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971518 2537 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971524 2537 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971527 2537 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971530 2537 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971533 2537 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971536 2537 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971538 2537 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971541 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971544 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971547 2537 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971550 2537 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971554 2537 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971561 2537 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971564 2537 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971567 2537 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971570 2537 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971573 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971576 2537 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 20:21:13.974169 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971579 2537 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971582 2537 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971585 2537 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971588 2537 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971590 2537 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971593 2537 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971596 2537 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971599 2537 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971601 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971604 2537 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971607 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971610 2537 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971612 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971615 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971617 2537 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971620 2537 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971622 2537 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971625 2537 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971628 2537 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971630 2537 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 20:21:13.974622 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971633 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971636 2537 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971639 2537 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971641 2537 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971644 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971647 2537 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971650 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971653 2537 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971656 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971659 2537 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971662 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971664 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971667 2537 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971670 2537 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971672 2537 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971675 2537 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971677 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971680 2537 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971683 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971685 2537 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 20:21:13.975130 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971688 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971690 2537 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971693 2537 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971695 2537 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971698 2537 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971700 2537 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971703 2537 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971705 2537 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971708 2537 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971711 2537 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971713 2537 feature_gate.go:328] unrecognized feature gate: Example Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971715 2537 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971718 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971721 2537 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971723 2537 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971726 2537 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971729 2537 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971731 2537 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971735 2537 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 20:21:13.975608 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971738 2537 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971742 2537 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971745 2537 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971747 2537 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971750 2537 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971754 2537 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971757 2537 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971760 2537 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971762 2537 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971765 2537 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971768 2537 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971770 2537 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971784 2537 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971787 2537 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971791 2537 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971795 2537 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971798 2537 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971801 2537 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971803 2537 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 20:21:13.976070 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971806 2537 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971809 2537 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:13.971811 2537 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.971816 2537 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.971945 2537 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.974010 2537 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.975073 2537 server.go:1019] "Starting client certificate rotation" Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.975185 2537 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 20:21:13.976518 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.975222 2537 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 20:21:13.977432 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.977418 2537 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 20:21:13.978903 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.978885 2537 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 20:21:13.986250 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.986231 2537 log.go:25] "Validated CRI v1 runtime API" Jul 30 20:21:13.988568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.988553 2537 log.go:25] "Validated CRI v1 image API" Jul 30 20:21:13.992170 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.992148 2537 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 20:21:13.994091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.994072 2537 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 922f728d-0d34-495e-9794-0ae2877e39d4:/dev/nvme0n1p4 cfd936fc-0924-49a0-82c1-124c1e4a4861:/dev/nvme0n1p3] Jul 30 20:21:13.994160 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:13.994091 2537 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 20:21:14.000420 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.000298 2537 manager.go:216] Machine: {Timestamp:2026-07-30 20:21:13.999191185 +0000 UTC m=+0.259503701 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101446 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b424a05355325f7e962b2d14f78b8 SystemUUID:ec2b424a-0535-5325-f7e9-62b2d14f78b8 BootID:ed01a918-6fd1-4e37-b38d-16fcd602ddc8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fd:44:74:30:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fd:44:74:30:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:fe:c2:88:3a:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 20:21:14.000420 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.000406 2537 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 20:21:14.000560 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.000490 2537 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 20:21:14.003006 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.002976 2537 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 20:21:14.003168 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.003010 2537 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 20:21:14.003642 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.003630 2537 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 20:21:14.003695 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.003648 2537 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 20:21:14.003695 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.003658 2537 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 20:21:14.003695 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.003677 2537 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 20:21:14.004828 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.004816 2537 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 20:21:14.009029 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.009015 2537 state_mem.go:36] "Initialized new in-memory state store" Jul 30 20:21:14.009165 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.009154 2537 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 20:21:14.009702 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.009692 2537 kubelet.go:486] "Attempting to sync node with API server" Jul 30 20:21:14.009751 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.009709 2537 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 20:21:14.009751 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.009729 2537 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 20:21:14.009751 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.009742 2537 kubelet.go:398] "Adding apiserver pod source" Jul 30 20:21:14.009895 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.009756 2537 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 20:21:14.010176 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.010163 2537 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 20:21:14.010239 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.010187 2537 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 20:21:14.011896 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.011881 2537 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 20:21:14.013160 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013147 2537 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 20:21:14.013207 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013178 2537 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 20:21:14.013737 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013727 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 20:21:14.013770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013743 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 20:21:14.013770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013749 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 20:21:14.013770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013754 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 20:21:14.013770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013765 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 20:21:14.013888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013792 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 20:21:14.013888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013802 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 20:21:14.013888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013807 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 20:21:14.013888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013814 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 20:21:14.013888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013819 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 20:21:14.013888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013829 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 20:21:14.013888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.013837 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 20:21:14.015039 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.015026 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 20:21:14.015100 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.015041 2537 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 20:21:14.018216 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.018201 2537 server.go:1268] "Started kubelet" Jul 30 20:21:14.018369 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.018338 2537 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 20:21:14.018464 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.018316 2537 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 20:21:14.018517 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.018501 2537 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 20:21:14.019117 ip-10-0-131-223 systemd[1]: Started Kubernetes Kubelet. Jul 30 20:21:14.020179 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.020001 2537 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 20:21:14.020792 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.020765 2537 server.go:310] "Adding debug handlers to kubelet server" Jul 30 20:21:14.020915 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.020896 2537 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 20:21:14.022540 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.022517 2537 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 20:21:14.022986 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.022939 2537 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 20:21:14.023092 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.023072 2537 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 20:21:14.025290 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.025276 2537 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 20:21:14.025290 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.025281 2537 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 20:21:14.025789 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.025600 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.026192 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026172 2537 factory.go:55] Registering systemd factory Jul 30 20:21:14.026192 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026181 2537 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 20:21:14.026316 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026199 2537 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 20:21:14.026316 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026225 2537 factory.go:223] Registration of the systemd container factory successfully Jul 30 20:21:14.026420 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026320 2537 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 20:21:14.026470 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.026423 2537 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 20:21:14.026522 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026497 2537 factory.go:153] Registering CRI-O factory Jul 30 20:21:14.026522 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026511 2537 factory.go:223] Registration of the crio container factory successfully Jul 30 20:21:14.026618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026564 2537 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 20:21:14.026618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026590 2537 factory.go:103] Registering Raw factory Jul 30 20:21:14.026618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026599 2537 reconstruct.go:97] "Volume reconstruction finished" Jul 30 20:21:14.026618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026606 2537 manager.go:1196] Started watching for new ooms in manager Jul 30 20:21:14.026618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.026612 2537 reconciler.go:29] "Reconciler: start to sync state" Jul 30 20:21:14.026822 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.025811 2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-223.ec2.internal.18c72ad5909b29eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-223.ec2.internal,UID:ip-10-0-131-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-223.ec2.internal,},FirstTimestamp:2026-07-30 20:21:14.018179563 +0000 UTC m=+0.278492081,LastTimestamp:2026-07-30 20:21:14.018179563 +0000 UTC m=+0.278492081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-223.ec2.internal,}" Jul 30 20:21:14.027555 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.027431 2537 manager.go:349] Starting recovery of all containers Jul 30 20:21:14.032247 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.032218 2537 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 20:21:14.032375 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.032349 2537 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 20:21:14.039858 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.039736 2537 manager.go:354] Recovery completed Jul 30 20:21:14.041525 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.041508 2537 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kssn6" Jul 30 20:21:14.043923 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.043911 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:14.045886 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.045869 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:14.045974 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.045902 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:14.045974 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.045922 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:14.046480 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.046466 2537 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 20:21:14.046480 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.046476 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:14.046562 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.046480 2537 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 20:21:14.046562 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.046561 2537 state_mem.go:36] "Initialized new in-memory state store" Jul 30 20:21:14.047165 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.047150 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:14.047240 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.047180 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:14.047240 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.047196 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:14.048571 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.048477 2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-223.ec2.internal.18c72ad59241ee30 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-223.ec2.internal,UID:ip-10-0-131-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-223.ec2.internal,},FirstTimestamp:2026-07-30 20:21:14.045886 +0000 UTC m=+0.306198521,LastTimestamp:2026-07-30 20:21:14.045886 +0000 UTC m=+0.306198521,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-223.ec2.internal,}" Jul 30 20:21:14.048712 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.048701 2537 policy_none.go:49] "None policy: Start" Jul 30 20:21:14.048752 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.048718 2537 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 20:21:14.048752 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.048728 2537 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 20:21:14.050716 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.050705 2537 policy_none.go:47] "Start" Jul 30 20:21:14.055453 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.055438 2537 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kssn6" Jul 30 20:21:14.059493 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.059423 2537 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-223.ec2.internal.18c72ad5924244a8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-223.ec2.internal,UID:ip-10-0-131-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-223.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-223.ec2.internal,},FirstTimestamp:2026-07-30 20:21:14.045908136 +0000 UTC m=+0.306220654,LastTimestamp:2026-07-30 20:21:14.045908136 +0000 UTC m=+0.306220654,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-223.ec2.internal,}" Jul 30 20:21:14.102418 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.102397 2537 manager.go:337] "Starting Device Plugin manager" Jul 30 20:21:14.102527 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.102432 2537 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 20:21:14.102527 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.102443 2537 server.go:87] "Starting device plugin registration server" Jul 30 20:21:14.102739 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.102726 2537 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 20:21:14.102796 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.102744 2537 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 20:21:14.102856 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.102827 2537 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 20:21:14.102954 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.102936 2537 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 20:21:14.102954 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.102948 2537 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 20:21:14.103417 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.103399 2537 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 20:21:14.103492 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.103455 2537 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.122108 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.122071 2537 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 20:21:14.129670 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.123200 2537 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 20:21:14.129670 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.123241 2537 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 20:21:14.129670 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.123264 2537 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 20:21:14.129670 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.123296 2537 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 20:21:14.129670 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.126164 2537 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:14.207355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.207249 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:14.208341 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.208319 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:14.208435 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.208356 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:14.208435 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.208372 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:14.208435 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.208400 2537 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.217441 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.217417 2537 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.217507 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.217446 2537 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-223.ec2.internal\": node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.223471 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.223442 2537 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal"] Jul 30 20:21:14.223541 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.223507 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:14.224997 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.224977 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:14.225101 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.225011 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:14.225101 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.225025 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:14.227622 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.227606 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:14.227716 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.227619 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d84a9cee39c2aef80bd2601e696693cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal\" (UID: \"d84a9cee39c2aef80bd2601e696693cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.227716 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.227649 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d84a9cee39c2aef80bd2601e696693cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal\" (UID: \"d84a9cee39c2aef80bd2601e696693cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.227716 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.227707 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.227844 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.227736 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:14.228334 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.228316 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:14.228396 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.228348 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:14.228396 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.228318 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:14.228396 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.228363 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:14.228396 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.228380 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:14.228396 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.228392 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:14.231444 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.231429 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.231510 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.231457 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:14.232188 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.232170 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:14.232295 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.232202 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:14.232295 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.232216 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:14.238093 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.238076 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.251826 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.251786 2537 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-223.ec2.internal\" not found" node="ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.256568 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.256548 2537 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-223.ec2.internal\" not found" node="ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.328018 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.327982 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d84a9cee39c2aef80bd2601e696693cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal\" (UID: \"d84a9cee39c2aef80bd2601e696693cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.328018 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.328017 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d84a9cee39c2aef80bd2601e696693cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal\" (UID: \"d84a9cee39c2aef80bd2601e696693cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.328180 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.328040 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a2f016900707da51a5c4bb3e93fff86-config\") pod \"kube-apiserver-proxy-ip-10-0-131-223.ec2.internal\" (UID: \"7a2f016900707da51a5c4bb3e93fff86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.328180 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.328072 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d84a9cee39c2aef80bd2601e696693cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal\" (UID: \"d84a9cee39c2aef80bd2601e696693cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.328180 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.328076 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d84a9cee39c2aef80bd2601e696693cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal\" (UID: \"d84a9cee39c2aef80bd2601e696693cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.338871 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.338842 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.428987 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.428954 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a2f016900707da51a5c4bb3e93fff86-config\") pod \"kube-apiserver-proxy-ip-10-0-131-223.ec2.internal\" (UID: \"7a2f016900707da51a5c4bb3e93fff86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.429133 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.428996 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a2f016900707da51a5c4bb3e93fff86-config\") pod \"kube-apiserver-proxy-ip-10-0-131-223.ec2.internal\" (UID: \"7a2f016900707da51a5c4bb3e93fff86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.439030 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.439000 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.539672 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.539584 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.554911 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.554883 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.559244 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.559226 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" Jul 30 20:21:14.640868 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.640825 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.740997 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.740958 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.841139 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.841028 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.860619 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.860589 2537 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:14.941433 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:14.941372 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:14.975201 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.975157 2537 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 20:21:14.975773 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.975359 2537 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 20:21:14.975773 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:14.975368 2537 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 20:21:15.041713 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:15.041681 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:15.059708 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.059667 2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 20:16:14 +0000 UTC" deadline="2028-03-21 02:17:46.18449628 +0000 UTC" Jul 30 20:21:15.059708 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.059703 2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14381h56m31.124797235s" Jul 30 20:21:15.142492 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:15.142394 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:15.189337 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:15.189300 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84a9cee39c2aef80bd2601e696693cf.slice/crio-c23d421552349b880a296b1f40b1b2339a9e241172ad29850dd9b044fee73828 WatchSource:0}: Error finding container c23d421552349b880a296b1f40b1b2339a9e241172ad29850dd9b044fee73828: Status 404 returned error can't find the container with id c23d421552349b880a296b1f40b1b2339a9e241172ad29850dd9b044fee73828 Jul 30 20:21:15.193561 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.193545 2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 20:21:15.243439 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:15.243379 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:15.259745 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.259719 2537 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 20:21:15.261132 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.261111 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientMemory" Jul 30 20:21:15.261249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.261151 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 20:21:15.261249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.261167 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeHasSufficientPID" Jul 30 20:21:15.344593 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:15.344548 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:15.445774 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:15.445687 2537 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-223.ec2.internal\" not found" Jul 30 20:21:15.448344 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.448319 2537 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:15.474209 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.474179 2537 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:15.526357 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.526313 2537 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" Jul 30 20:21:15.532339 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:15.532311 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2f016900707da51a5c4bb3e93fff86.slice/crio-56c43bf2d00058d29a06de6074e64ae51296332d11d9935b49cbd8002a8a4889 WatchSource:0}: Error finding container 56c43bf2d00058d29a06de6074e64ae51296332d11d9935b49cbd8002a8a4889: Status 404 returned error can't find the container with id 56c43bf2d00058d29a06de6074e64ae51296332d11d9935b49cbd8002a8a4889 Jul 30 20:21:15.538812 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.538793 2537 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 20:21:15.539693 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.539676 2537 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" Jul 30 20:21:15.546068 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.546051 2537 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 20:21:15.979638 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:15.979606 2537 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:16.012529 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.012490 2537 apiserver.go:52] "Watching apiserver" Jul 30 20:21:16.020360 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.020331 2537 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 20:21:16.022086 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.022056 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-b7qhp","openshift-dns/node-resolver-2w9bg","openshift-multus/multus-s7jf2","openshift-network-diagnostics/network-check-target-fnq9v","openshift-network-operator/iptables-alerter-9ttb9","kube-system/konnectivity-agent-97pmk","kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal","openshift-image-registry/node-ca-b9lwj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal","openshift-multus/multus-additional-cni-plugins-k5h49","openshift-multus/network-metrics-daemon-q2k9d","openshift-ovn-kubernetes/ovnkube-node-kdnvq","kube-system/global-pull-secret-syncer-gr4lp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs"] Jul 30 20:21:16.025082 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.025056 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.025995 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.025977 2537 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 20:21:16.027507 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.027234 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.027507 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.027311 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:16.028343 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.028175 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 20:21:16.029285 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.028528 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 20:21:16.029285 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.028671 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 20:21:16.029285 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.029042 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 20:21:16.029285 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.029134 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-626sr\"" Jul 30 20:21:16.030248 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.029747 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.030685 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.030502 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 20:21:16.033091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.032450 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:16.033091 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.032514 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:16.033091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.032691 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 20:21:16.033091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.032928 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rqbh\"" Jul 30 20:21:16.035424 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.034966 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.037403 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.037381 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:16.037512 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.037494 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 20:21:16.037718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.037701 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2pxpq\"" Jul 30 20:21:16.037781 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.037700 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:16.039263 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039237 2537 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039480 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039647 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-system-cni-dir\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039680 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-cnibin\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039706 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039726 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039752 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039783 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-kubelet-config\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039806 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-os-release\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039831 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-cni-binary-copy\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039888 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdss\" (UniqueName: \"kubernetes.io/projected/3ce791b2-1232-4fbe-9ffe-523b7263a619-kube-api-access-5vdss\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039917 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-dbus\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.040297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.039950 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.042121 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.042093 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 20:21:16.042779 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.042614 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.043802 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.043378 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-955jf\"" Jul 30 20:21:16.043969 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.043947 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 20:21:16.044986 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.044965 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.047625 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.046055 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 20:21:16.047724 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.046844 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 20:21:16.047785 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.046939 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8npwj\"" Jul 30 20:21:16.048320 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.047056 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 20:21:16.049749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.049713 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zslzk\"" Jul 30 20:21:16.049951 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.049933 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 20:21:16.050191 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.050161 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 20:21:16.050359 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.050344 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 20:21:16.050836 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.050631 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.053146 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.053045 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 20:21:16.053146 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.053096 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 20:21:16.054087 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.053495 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.054246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.054188 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fq9fk\"" Jul 30 20:21:16.058336 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.056980 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:21:16.058336 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.058004 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:16.058336 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.058092 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:16.058794 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.058774 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pwxlr\"" Jul 30 20:21:16.059661 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.059206 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 20:21:16.061249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.060308 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.062594 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.062565 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 20:21:16.062936 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.062901 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52d8c\"" Jul 30 20:21:16.063550 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.063539 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 20:21:16.063802 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.063784 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 20:21:16.064485 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.064144 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 20:21:16.064485 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.064377 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 20:21:16.064664 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.064597 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 20:21:16.066837 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.066814 2537 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-785mb" Jul 30 20:21:16.076290 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.076267 2537 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-785mb" Jul 30 20:21:16.127878 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.127844 2537 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 20:21:16.130987 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.130931 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" event={"ID":"7a2f016900707da51a5c4bb3e93fff86","Type":"ContainerStarted","Data":"56c43bf2d00058d29a06de6074e64ae51296332d11d9935b49cbd8002a8a4889"} Jul 30 20:21:16.133138 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.133106 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" event={"ID":"d84a9cee39c2aef80bd2601e696693cf","Type":"ContainerStarted","Data":"c23d421552349b880a296b1f40b1b2339a9e241172ad29850dd9b044fee73828"} Jul 30 20:21:16.140453 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140420 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.140613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140472 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-kubelet\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.140613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140523 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmzv\" (UniqueName: \"kubernetes.io/projected/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-kube-api-access-plmzv\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:16.140613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140550 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-sys\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.140613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140584 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovnkube-config\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140610 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-multus-certs\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140637 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-sys-fs\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140661 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccj6\" (UniqueName: \"kubernetes.io/projected/88b6c1c4-8292-4249-81ff-7c1e860c75f7-kube-api-access-7ccj6\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140686 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-run-netns\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140710 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140735 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-socket-dir-parent\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140763 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-netns\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.140829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140808 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-device-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140860 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-serviceca\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140885 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysctl-d\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140908 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-run\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140947 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-os-release\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.140972 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-etc-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141001 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141030 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9-agent-certs\") pod \"konnectivity-agent-97pmk\" (UID: \"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9\") " pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141067 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94f79733-5c93-41b4-8960-4ffd7f4f0592-tmp\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141093 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-kubelet\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141102 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141117 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-node-log\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141144 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-cni-netd\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141179 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.141246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141219 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-cni-multus\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141300 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vgdd\" (UniqueName: \"kubernetes.io/projected/fa158ca8-9798-4ad3-b302-a69eb51e1295-kube-api-access-5vgdd\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141322 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6hq\" (UniqueName: \"kubernetes.io/projected/fc85afc9-0be3-43a2-b2b0-96a4143daa28-kube-api-access-hm6hq\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141338 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-os-release\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141344 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-registration-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141364 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-daemon-config\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141397 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-dbus\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141437 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-kubelet-config\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141495 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141521 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-cni-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141526 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-kubelet-config\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141561 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysctl-conf\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141564 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-dbus\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141600 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-cni-binary-copy\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141637 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-cni-binary-copy\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141715 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141743 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-socket-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.141845 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141761 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-host\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141808 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-log-socket\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141836 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-cnibin\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141859 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141875 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmld\" (UniqueName: \"kubernetes.io/projected/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-kube-api-access-sgmld\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141910 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-lib-modules\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141937 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141946 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-cnibin\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141961 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhs55\" (UniqueName: \"kubernetes.io/projected/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-kube-api-access-mhs55\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.141985 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa158ca8-9798-4ad3-b302-a69eb51e1295-hosts-file\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142013 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142040 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-kubernetes\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142078 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmjq\" (UniqueName: \"kubernetes.io/projected/94f79733-5c93-41b4-8960-4ffd7f4f0592-kube-api-access-2fmjq\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142082 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-cni-binary-copy\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142095 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-cni-bin\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142120 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-etc-kubernetes\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.142718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142143 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-tuned\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142167 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-var-lib-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142187 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovnkube-script-lib\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142208 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysconfig\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142236 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-systemd\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142263 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-system-cni-dir\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142305 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/680f84f3-d507-47cf-bc45-157fb05484f4-iptables-alerter-script\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142319 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ce791b2-1232-4fbe-9ffe-523b7263a619-system-cni-dir\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142353 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-systemd\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142386 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdss\" (UniqueName: \"kubernetes.io/projected/3ce791b2-1232-4fbe-9ffe-523b7263a619-kube-api-access-5vdss\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142412 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-system-cni-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142438 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-os-release\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142466 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/680f84f3-d507-47cf-bc45-157fb05484f4-host-slash\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142508 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa158ca8-9798-4ad3-b302-a69eb51e1295-tmp-dir\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142530 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-modprobe-d\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142553 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-var-lib-kubelet\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142578 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-systemd-units\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.143620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.142613 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-cnibin\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143145 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9-konnectivity-ca\") pod \"konnectivity-agent-97pmk\" (UID: \"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9\") " pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143173 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-k8s-cni-cncf-io\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143230 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-conf-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143269 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143300 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143325 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-slash\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143344 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-cni-bin\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143360 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-hostroot\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143384 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97d42\" (UniqueName: \"kubernetes.io/projected/680f84f3-d507-47cf-bc45-157fb05484f4-kube-api-access-97d42\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143410 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143433 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-host\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.143453 2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.143543 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret podName:0e4765f1-4ab8-4d6c-8f93-0ee04514feb1 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:16.643507869 +0000 UTC m=+2.903820380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret") pod "global-pull-secret-syncer-gr4lp" (UID: "0e4765f1-4ab8-4d6c-8f93-0ee04514feb1") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143457 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-ovn\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143584 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-env-overrides\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.144568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143611 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovn-node-metrics-cert\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.145432 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.143914 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ce791b2-1232-4fbe-9ffe-523b7263a619-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.149578 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.149538 2537 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 20:21:16.153903 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.153872 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdss\" (UniqueName: \"kubernetes.io/projected/3ce791b2-1232-4fbe-9ffe-523b7263a619-kube-api-access-5vdss\") pod \"multus-additional-cni-plugins-k5h49\" (UID: \"3ce791b2-1232-4fbe-9ffe-523b7263a619\") " pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.245081 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245001 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-kubelet\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.245081 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245037 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-node-log\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.245081 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245056 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-cni-netd\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.245081 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245076 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245101 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-cni-multus\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245119 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vgdd\" (UniqueName: \"kubernetes.io/projected/fa158ca8-9798-4ad3-b302-a69eb51e1295-kube-api-access-5vgdd\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245136 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6hq\" (UniqueName: \"kubernetes.io/projected/fc85afc9-0be3-43a2-b2b0-96a4143daa28-kube-api-access-hm6hq\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245154 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-registration-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245180 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-daemon-config\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245208 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-cni-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245225 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysctl-conf\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245241 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-cni-binary-copy\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245256 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245272 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-socket-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245295 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-host\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245310 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-log-socket\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245327 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245342 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmld\" (UniqueName: \"kubernetes.io/projected/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-kube-api-access-sgmld\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.245376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245362 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-lib-modules\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245385 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245412 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhs55\" (UniqueName: \"kubernetes.io/projected/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-kube-api-access-mhs55\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245436 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa158ca8-9798-4ad3-b302-a69eb51e1295-hosts-file\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245455 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245476 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-kubernetes\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245494 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmjq\" (UniqueName: \"kubernetes.io/projected/94f79733-5c93-41b4-8960-4ffd7f4f0592-kube-api-access-2fmjq\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245510 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-cni-bin\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245527 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-etc-kubernetes\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245547 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-tuned\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245573 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-var-lib-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245588 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovnkube-script-lib\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245608 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysconfig\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245629 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-systemd\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245652 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/680f84f3-d507-47cf-bc45-157fb05484f4-iptables-alerter-script\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245672 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-systemd\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245690 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-system-cni-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246001 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245707 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-os-release\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245728 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/680f84f3-d507-47cf-bc45-157fb05484f4-host-slash\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245750 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa158ca8-9798-4ad3-b302-a69eb51e1295-tmp-dir\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245777 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-modprobe-d\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245800 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-var-lib-kubelet\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245821 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-systemd-units\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245845 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-cnibin\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245889 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9-konnectivity-ca\") pod \"konnectivity-agent-97pmk\" (UID: \"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9\") " pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245902 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-node-log\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245919 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-k8s-cni-cncf-io\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245943 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-conf-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.245975 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-cni-netd\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246003 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-slash\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246032 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-cni-bin\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246059 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-hostroot\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246071 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-cni-multus\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246092 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97d42\" (UniqueName: \"kubernetes.io/projected/680f84f3-d507-47cf-bc45-157fb05484f4-kube-api-access-97d42\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246115 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-kubelet\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.246948 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246136 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246162 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-host\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246163 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa158ca8-9798-4ad3-b302-a69eb51e1295-hosts-file\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246203 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-ovn\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246220 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-host\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246233 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-env-overrides\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246263 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovn-node-metrics-cert\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246275 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysconfig\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246292 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-kubelet\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246319 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-systemd\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246384 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-systemd\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246425 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246436 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-system-cni-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246515 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-os-release\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246560 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/680f84f3-d507-47cf-bc45-157fb05484f4-host-slash\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246608 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-registration-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246683 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-kubernetes\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246928 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-cni-bin\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.247925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246940 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa158ca8-9798-4ad3-b302-a69eb51e1295-tmp-dir\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246981 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-etc-kubernetes\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247121 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-modprobe-d\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247177 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-var-lib-kubelet\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247195 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-daemon-config\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247224 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-systemd-units\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247266 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247285 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-cni-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247316 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-cnibin\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247372 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-ovn\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247413 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-log-socket\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247429 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-var-lib-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246030 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247487 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247539 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-host\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247539 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-slash\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247420 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysctl-conf\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247583 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-cni-bin\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.248968 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247875 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9-konnectivity-ca\") pod \"konnectivity-agent-97pmk\" (UID: \"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9\") " pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247934 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-hostroot\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.247996 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/680f84f3-d507-47cf-bc45-157fb05484f4-iptables-alerter-script\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.246321 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmzv\" (UniqueName: \"kubernetes.io/projected/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-kube-api-access-plmzv\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.248239 2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248288 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovnkube-script-lib\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.248300 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs podName:0de7db11-2876-47d9-bbd3-c2a8faf8bf8d nodeName:}" failed. No retries permitted until 2026-07-30 20:21:16.748281233 +0000 UTC m=+3.008593760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs") pod "network-metrics-daemon-q2k9d" (UID: "0de7db11-2876-47d9-bbd3-c2a8faf8bf8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248461 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-cni-binary-copy\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248468 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-k8s-cni-cncf-io\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248505 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-var-lib-kubelet\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248530 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-conf-dir\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248576 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-lib-modules\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248588 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-sys\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248619 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovnkube-config\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248619 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-socket-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248648 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-multus-certs\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248671 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-sys\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.249945 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248758 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-sys-fs\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248802 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccj6\" (UniqueName: \"kubernetes.io/projected/88b6c1c4-8292-4249-81ff-7c1e860c75f7-kube-api-access-7ccj6\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248844 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-run-netns\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248895 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248932 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-host-run-netns\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.248985 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-socket-dir-parent\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249016 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-netns\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249028 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-multus-socket-dir-parent\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249046 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-env-overrides\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249058 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-sys-fs\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249107 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-device-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249136 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-serviceca\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249141 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88b6c1c4-8292-4249-81ff-7c1e860c75f7-device-dir\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249180 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovnkube-config\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249187 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-netns\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249221 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-host-run-multus-certs\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249237 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysctl-d\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249240 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-run-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.250977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249286 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-run\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249324 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-etc-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249338 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-sysctl-d\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249363 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9-agent-certs\") pod \"konnectivity-agent-97pmk\" (UID: \"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9\") " pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249369 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94f79733-5c93-41b4-8960-4ffd7f4f0592-run\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249406 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94f79733-5c93-41b4-8960-4ffd7f4f0592-tmp\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249429 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc85afc9-0be3-43a2-b2b0-96a4143daa28-etc-openvswitch\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.249552 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-serviceca\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.252145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.251167 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc85afc9-0be3-43a2-b2b0-96a4143daa28-ovn-node-metrics-cert\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.253743 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.253719 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94f79733-5c93-41b4-8960-4ffd7f4f0592-etc-tuned\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.255603 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.255556 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94f79733-5c93-41b4-8960-4ffd7f4f0592-tmp\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.256257 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.256232 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9-agent-certs\") pod \"konnectivity-agent-97pmk\" (UID: \"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9\") " pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.257578 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.257541 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6hq\" (UniqueName: \"kubernetes.io/projected/fc85afc9-0be3-43a2-b2b0-96a4143daa28-kube-api-access-hm6hq\") pod \"ovnkube-node-kdnvq\" (UID: \"fc85afc9-0be3-43a2-b2b0-96a4143daa28\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.258027 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.257984 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vgdd\" (UniqueName: \"kubernetes.io/projected/fa158ca8-9798-4ad3-b302-a69eb51e1295-kube-api-access-5vgdd\") pod \"node-resolver-2w9bg\" (UID: \"fa158ca8-9798-4ad3-b302-a69eb51e1295\") " pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.259467 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.259439 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhs55\" (UniqueName: \"kubernetes.io/projected/67bcbfeb-4460-4bc4-bc79-0cfcc04f0936-kube-api-access-mhs55\") pod \"multus-s7jf2\" (UID: \"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936\") " pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.260014 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.259993 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmzv\" (UniqueName: \"kubernetes.io/projected/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-kube-api-access-plmzv\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:16.261363 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.261320 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmjq\" (UniqueName: \"kubernetes.io/projected/94f79733-5c93-41b4-8960-4ffd7f4f0592-kube-api-access-2fmjq\") pod \"tuned-b7qhp\" (UID: \"94f79733-5c93-41b4-8960-4ffd7f4f0592\") " pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.262572 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.262307 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:16.262572 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.262331 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:16.262572 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.262347 2537 projected.go:196] Error preparing data for projected volume kube-api-access-mh8l9 for pod openshift-network-diagnostics/network-check-target-fnq9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:16.262572 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.262410 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9 podName:7f0a7e93-0b67-40fe-a946-d8fc39a126c6 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:16.76238959 +0000 UTC m=+3.022702102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mh8l9" (UniqueName: "kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9") pod "network-check-target-fnq9v" (UID: "7f0a7e93-0b67-40fe-a946-d8fc39a126c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:16.264995 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.264953 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmld\" (UniqueName: \"kubernetes.io/projected/16f37bef-efea-4c62-b4ca-fcb69b4ecaad-kube-api-access-sgmld\") pod \"node-ca-b9lwj\" (UID: \"16f37bef-efea-4c62-b4ca-fcb69b4ecaad\") " pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.266445 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.266398 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97d42\" (UniqueName: \"kubernetes.io/projected/680f84f3-d507-47cf-bc45-157fb05484f4-kube-api-access-97d42\") pod \"iptables-alerter-9ttb9\" (UID: \"680f84f3-d507-47cf-bc45-157fb05484f4\") " pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.267322 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.267295 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccj6\" (UniqueName: \"kubernetes.io/projected/88b6c1c4-8292-4249-81ff-7c1e860c75f7-kube-api-access-7ccj6\") pod \"aws-ebs-csi-driver-node-7qzgs\" (UID: \"88b6c1c4-8292-4249-81ff-7c1e860c75f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.348615 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.348575 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k5h49" Jul 30 20:21:16.361074 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.361012 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s7jf2" Jul 30 20:21:16.370058 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.370034 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ttb9" Jul 30 20:21:16.379131 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.379069 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:16.387192 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.387167 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" Jul 30 20:21:16.400128 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.400049 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b9lwj" Jul 30 20:21:16.407133 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.407104 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2w9bg" Jul 30 20:21:16.419248 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.419219 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" Jul 30 20:21:16.424431 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.424252 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:16.652544 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.652458 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:16.652724 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.652660 2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:16.652790 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.652757 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret podName:0e4765f1-4ab8-4d6c-8f93-0ee04514feb1 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:17.652722277 +0000 UTC m=+3.913034795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret") pod "global-pull-secret-syncer-gr4lp" (UID: "0e4765f1-4ab8-4d6c-8f93-0ee04514feb1") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:16.753421 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.753377 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:16.753619 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.753573 2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:16.753680 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.753664 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs podName:0de7db11-2876-47d9-bbd3-c2a8faf8bf8d nodeName:}" failed. No retries permitted until 2026-07-30 20:21:17.753640941 +0000 UTC m=+4.013953452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs") pod "network-metrics-daemon-q2k9d" (UID: "0de7db11-2876-47d9-bbd3-c2a8faf8bf8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:16.853967 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:16.853925 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:16.854143 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.854105 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:16.854143 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.854134 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:16.854143 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.854145 2537 projected.go:196] Error preparing data for projected volume kube-api-access-mh8l9 for pod openshift-network-diagnostics/network-check-target-fnq9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:16.854247 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:16.854205 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9 podName:7f0a7e93-0b67-40fe-a946-d8fc39a126c6 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:17.854186041 +0000 UTC m=+4.114498546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh8l9" (UniqueName: "kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9") pod "network-check-target-fnq9v" (UID: "7f0a7e93-0b67-40fe-a946-d8fc39a126c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:17.080768 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:17.080631 2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 20:16:16 +0000 UTC" deadline="2028-04-07 05:10:08.600627226 +0000 UTC" Jul 30 20:21:17.080768 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:17.080692 2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14792h48m51.519939978s" Jul 30 20:21:17.186490 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.186460 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc85afc9_0be3_43a2_b2b0_96a4143daa28.slice/crio-4490cde9b899f599240e2a4a68093b15f0abd3dcb13353bd2ed5032b9e923900 WatchSource:0}: Error finding container 4490cde9b899f599240e2a4a68093b15f0abd3dcb13353bd2ed5032b9e923900: Status 404 returned error can't find the container with id 4490cde9b899f599240e2a4a68093b15f0abd3dcb13353bd2ed5032b9e923900 Jul 30 20:21:17.187569 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.187540 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680f84f3_d507_47cf_bc45_157fb05484f4.slice/crio-b918e16e32770104143bca6eaeaa9e7e7774e0bf6a974f1eb9f9504a0303c655 WatchSource:0}: Error finding container b918e16e32770104143bca6eaeaa9e7e7774e0bf6a974f1eb9f9504a0303c655: Status 404 returned error can't find the container with id b918e16e32770104143bca6eaeaa9e7e7774e0bf6a974f1eb9f9504a0303c655 Jul 30 20:21:17.191588 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.191518 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c15f8d3_eeb8_4d1e_8705_ac2ae2344cb9.slice/crio-a78a8a1595dbc891771578568ae1a1fda9f9df5be65f33efcdbf9b9fdff61682 WatchSource:0}: Error finding container a78a8a1595dbc891771578568ae1a1fda9f9df5be65f33efcdbf9b9fdff61682: Status 404 returned error can't find the container with id a78a8a1595dbc891771578568ae1a1fda9f9df5be65f33efcdbf9b9fdff61682 Jul 30 20:21:17.212883 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.212844 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa158ca8_9798_4ad3_b302_a69eb51e1295.slice/crio-3dd0ecbcfd82c543b36e0e718558f64e9072540bb4b7f392a3b3cd94ebf8403d WatchSource:0}: Error finding container 3dd0ecbcfd82c543b36e0e718558f64e9072540bb4b7f392a3b3cd94ebf8403d: Status 404 returned error can't find the container with id 3dd0ecbcfd82c543b36e0e718558f64e9072540bb4b7f392a3b3cd94ebf8403d Jul 30 20:21:17.213691 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.213666 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce791b2_1232_4fbe_9ffe_523b7263a619.slice/crio-30837a6b793fc34bd15004c72575cb6886dd4e1776c67ea126ee9c29203c61ec WatchSource:0}: Error finding container 30837a6b793fc34bd15004c72575cb6886dd4e1776c67ea126ee9c29203c61ec: Status 404 returned error can't find the container with id 30837a6b793fc34bd15004c72575cb6886dd4e1776c67ea126ee9c29203c61ec Jul 30 20:21:17.215126 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.215082 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b6c1c4_8292_4249_81ff_7c1e860c75f7.slice/crio-ec9aeb63a8016ad4e48a0a05c8ddca4114798ca9cabab3a3c931c1d25779571d WatchSource:0}: Error finding container ec9aeb63a8016ad4e48a0a05c8ddca4114798ca9cabab3a3c931c1d25779571d: Status 404 returned error can't find the container with id ec9aeb63a8016ad4e48a0a05c8ddca4114798ca9cabab3a3c931c1d25779571d Jul 30 20:21:17.216241 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.216212 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bcbfeb_4460_4bc4_bc79_0cfcc04f0936.slice/crio-46bba563463a1e37bb8ec89db17b8ee17e58d5e2401f8d5f2c659cb45e980032 WatchSource:0}: Error finding container 46bba563463a1e37bb8ec89db17b8ee17e58d5e2401f8d5f2c659cb45e980032: Status 404 returned error can't find the container with id 46bba563463a1e37bb8ec89db17b8ee17e58d5e2401f8d5f2c659cb45e980032 Jul 30 20:21:17.217896 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.217439 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f79733_5c93_41b4_8960_4ffd7f4f0592.slice/crio-fc4a7d2c047d76d7452dde176502351a32efe70d16c595c9029a89294ab8122e WatchSource:0}: Error finding container fc4a7d2c047d76d7452dde176502351a32efe70d16c595c9029a89294ab8122e: Status 404 returned error can't find the container with id fc4a7d2c047d76d7452dde176502351a32efe70d16c595c9029a89294ab8122e Jul 30 20:21:17.218702 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:17.218642 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f37bef_efea_4c62_b4ca_fcb69b4ecaad.slice/crio-93222be6a2d338eb45dcc772ff1fedd6a6692ca688ce210b2bf9b8e1dfc0557e WatchSource:0}: Error finding container 93222be6a2d338eb45dcc772ff1fedd6a6692ca688ce210b2bf9b8e1dfc0557e: Status 404 returned error can't find the container with id 93222be6a2d338eb45dcc772ff1fedd6a6692ca688ce210b2bf9b8e1dfc0557e Jul 30 20:21:17.660721 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:17.660674 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:17.660904 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.660877 2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:17.660968 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.660946 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret podName:0e4765f1-4ab8-4d6c-8f93-0ee04514feb1 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:19.660925078 +0000 UTC m=+5.921237588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret") pod "global-pull-secret-syncer-gr4lp" (UID: "0e4765f1-4ab8-4d6c-8f93-0ee04514feb1") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:17.762004 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:17.761902 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:17.762172 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.762060 2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:17.762172 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.762132 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs podName:0de7db11-2876-47d9-bbd3-c2a8faf8bf8d nodeName:}" failed. No retries permitted until 2026-07-30 20:21:19.762111297 +0000 UTC m=+6.022423817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs") pod "network-metrics-daemon-q2k9d" (UID: "0de7db11-2876-47d9-bbd3-c2a8faf8bf8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:17.799465 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:17.799420 2537 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 20:21:17.863295 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:17.863250 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:17.863553 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.863468 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:17.863553 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.863491 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:17.863553 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.863506 2537 projected.go:196] Error preparing data for projected volume kube-api-access-mh8l9 for pod openshift-network-diagnostics/network-check-target-fnq9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:17.863739 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:17.863577 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9 podName:7f0a7e93-0b67-40fe-a946-d8fc39a126c6 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:19.863552775 +0000 UTC m=+6.123865305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh8l9" (UniqueName: "kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9") pod "network-check-target-fnq9v" (UID: "7f0a7e93-0b67-40fe-a946-d8fc39a126c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:18.081829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.081678 2537 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 20:16:16 +0000 UTC" deadline="2028-05-14 07:41:01.891411744 +0000 UTC" Jul 30 20:21:18.081829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.081725 2537 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15683h19m43.809692167s" Jul 30 20:21:18.126764 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.126037 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:18.126764 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:18.126180 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:18.126764 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.126045 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:18.126764 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:18.126303 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:18.126764 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.126045 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:18.126764 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:18.126384 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:18.141442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.141392 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ttb9" event={"ID":"680f84f3-d507-47cf-bc45-157fb05484f4","Type":"ContainerStarted","Data":"b918e16e32770104143bca6eaeaa9e7e7774e0bf6a974f1eb9f9504a0303c655"} Jul 30 20:21:18.144038 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.144009 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" event={"ID":"7a2f016900707da51a5c4bb3e93fff86","Type":"ContainerStarted","Data":"74c2effab41cd13318f0066151c5d18c1a036d895cff575135ad131ca4614592"} Jul 30 20:21:18.148632 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.148607 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2w9bg" event={"ID":"fa158ca8-9798-4ad3-b302-a69eb51e1295","Type":"ContainerStarted","Data":"3dd0ecbcfd82c543b36e0e718558f64e9072540bb4b7f392a3b3cd94ebf8403d"} Jul 30 20:21:18.153065 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.153043 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerStarted","Data":"30837a6b793fc34bd15004c72575cb6886dd4e1776c67ea126ee9c29203c61ec"} Jul 30 20:21:18.155289 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.155263 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" event={"ID":"88b6c1c4-8292-4249-81ff-7c1e860c75f7","Type":"ContainerStarted","Data":"ec9aeb63a8016ad4e48a0a05c8ddca4114798ca9cabab3a3c931c1d25779571d"} Jul 30 20:21:18.163696 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.163665 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-97pmk" event={"ID":"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9","Type":"ContainerStarted","Data":"a78a8a1595dbc891771578568ae1a1fda9f9df5be65f33efcdbf9b9fdff61682"} Jul 30 20:21:18.166105 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.166004 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"4490cde9b899f599240e2a4a68093b15f0abd3dcb13353bd2ed5032b9e923900"} Jul 30 20:21:18.170417 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.169623 2537 generic.go:356] "Generic (PLEG): container finished" podID="d84a9cee39c2aef80bd2601e696693cf" containerID="d49de57fe26e765513cab057e9b83e1aa0898fb4e5c10b5a9904732612059297" exitCode=0 Jul 30 20:21:18.170417 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.170227 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" event={"ID":"d84a9cee39c2aef80bd2601e696693cf","Type":"ContainerDied","Data":"d49de57fe26e765513cab057e9b83e1aa0898fb4e5c10b5a9904732612059297"} Jul 30 20:21:18.172814 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.172749 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b9lwj" event={"ID":"16f37bef-efea-4c62-b4ca-fcb69b4ecaad","Type":"ContainerStarted","Data":"93222be6a2d338eb45dcc772ff1fedd6a6692ca688ce210b2bf9b8e1dfc0557e"} Jul 30 20:21:18.176029 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.175844 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s7jf2" event={"ID":"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936","Type":"ContainerStarted","Data":"46bba563463a1e37bb8ec89db17b8ee17e58d5e2401f8d5f2c659cb45e980032"} Jul 30 20:21:18.182314 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.182284 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" event={"ID":"94f79733-5c93-41b4-8960-4ffd7f4f0592","Type":"ContainerStarted","Data":"fc4a7d2c047d76d7452dde176502351a32efe70d16c595c9029a89294ab8122e"} Jul 30 20:21:18.185009 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:18.184947 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-223.ec2.internal" podStartSLOduration=3.184929318 podStartE2EDuration="3.184929318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:18.15932032 +0000 UTC m=+4.419632863" watchObservedRunningTime="2026-07-30 20:21:18.184929318 +0000 UTC m=+4.445241849" Jul 30 20:21:19.210753 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:19.210670 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" event={"ID":"d84a9cee39c2aef80bd2601e696693cf","Type":"ContainerStarted","Data":"f39929986553049be59f1f037f6aa4f8370d0dfc1f14f8a8d9f0bdb4ff4692df"} Jul 30 20:21:19.678691 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:19.678475 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:19.678691 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.678659 2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:19.679122 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.678744 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret podName:0e4765f1-4ab8-4d6c-8f93-0ee04514feb1 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:23.678720362 +0000 UTC m=+9.939032871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret") pod "global-pull-secret-syncer-gr4lp" (UID: "0e4765f1-4ab8-4d6c-8f93-0ee04514feb1") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:19.779328 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:19.779241 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:19.779526 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.779437 2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:19.779526 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.779508 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs podName:0de7db11-2876-47d9-bbd3-c2a8faf8bf8d nodeName:}" failed. No retries permitted until 2026-07-30 20:21:23.779487479 +0000 UTC m=+10.039800011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs") pod "network-metrics-daemon-q2k9d" (UID: "0de7db11-2876-47d9-bbd3-c2a8faf8bf8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:19.881073 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:19.880236 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:19.881073 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.880523 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:19.881073 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.880547 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:19.881073 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.880561 2537 projected.go:196] Error preparing data for projected volume kube-api-access-mh8l9 for pod openshift-network-diagnostics/network-check-target-fnq9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:19.881073 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:19.880651 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9 podName:7f0a7e93-0b67-40fe-a946-d8fc39a126c6 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:23.880609852 +0000 UTC m=+10.140922385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh8l9" (UniqueName: "kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9") pod "network-check-target-fnq9v" (UID: "7f0a7e93-0b67-40fe-a946-d8fc39a126c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:20.125068 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:20.124980 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:20.125251 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:20.125124 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:20.125600 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:20.125548 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:20.125705 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:20.125664 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:20.125869 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:20.125838 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:20.125952 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:20.125928 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:22.124604 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:22.124093 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:22.124604 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:22.124234 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:22.124604 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:22.124596 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:22.125185 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:22.124693 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:22.125185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:22.124745 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:22.125185 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:22.124837 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:23.712482 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:23.711826 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:23.712482 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.712040 2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:23.712482 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.712103 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret podName:0e4765f1-4ab8-4d6c-8f93-0ee04514feb1 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:31.712085417 +0000 UTC m=+17.972397922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret") pod "global-pull-secret-syncer-gr4lp" (UID: "0e4765f1-4ab8-4d6c-8f93-0ee04514feb1") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:23.813208 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:23.813169 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:23.813400 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.813333 2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:23.813464 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.813403 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs podName:0de7db11-2876-47d9-bbd3-c2a8faf8bf8d nodeName:}" failed. No retries permitted until 2026-07-30 20:21:31.813384149 +0000 UTC m=+18.073696657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs") pod "network-metrics-daemon-q2k9d" (UID: "0de7db11-2876-47d9-bbd3-c2a8faf8bf8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:23.913913 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:23.913876 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:23.914151 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.914098 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:23.914151 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.914125 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:23.914151 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.914138 2537 projected.go:196] Error preparing data for projected volume kube-api-access-mh8l9 for pod openshift-network-diagnostics/network-check-target-fnq9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:23.914331 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:23.914201 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9 podName:7f0a7e93-0b67-40fe-a946-d8fc39a126c6 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:31.914183422 +0000 UTC m=+18.174495946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh8l9" (UniqueName: "kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9") pod "network-check-target-fnq9v" (UID: "7f0a7e93-0b67-40fe-a946-d8fc39a126c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:24.124574 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:24.124493 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:24.124574 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:24.124541 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:24.124794 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:24.124623 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:24.124794 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:24.124652 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:24.124794 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:24.124678 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:24.124794 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:24.124765 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:26.124429 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:26.124391 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:26.124937 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:26.124538 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:26.125011 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:26.124969 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:26.125100 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:26.125075 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:26.125161 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:26.125146 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:26.125231 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:26.125213 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:28.123770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:28.123732 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:28.124225 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:28.123850 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:28.124225 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:28.123739 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:28.124225 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:28.123882 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:28.124225 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:28.124000 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:28.124225 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:28.124088 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:30.124472 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:30.124439 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:30.124954 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:30.124439 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:30.124954 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:30.124571 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:30.124954 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:30.124453 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:30.124954 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:30.124690 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:30.124954 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:30.124792 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:31.771055 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:31.771014 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:31.771538 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.771137 2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:31.771538 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.771215 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret podName:0e4765f1-4ab8-4d6c-8f93-0ee04514feb1 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:47.771196594 +0000 UTC m=+34.031509118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret") pod "global-pull-secret-syncer-gr4lp" (UID: "0e4765f1-4ab8-4d6c-8f93-0ee04514feb1") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:31.871880 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:31.871841 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:31.872037 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.871991 2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:31.872089 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.872060 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs podName:0de7db11-2876-47d9-bbd3-c2a8faf8bf8d nodeName:}" failed. No retries permitted until 2026-07-30 20:21:47.872040473 +0000 UTC m=+34.132352993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs") pod "network-metrics-daemon-q2k9d" (UID: "0de7db11-2876-47d9-bbd3-c2a8faf8bf8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:31.973003 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:31.972968 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:31.973138 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.973100 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:31.973138 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.973116 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:31.973138 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.973128 2537 projected.go:196] Error preparing data for projected volume kube-api-access-mh8l9 for pod openshift-network-diagnostics/network-check-target-fnq9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:31.973236 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:31.973181 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9 podName:7f0a7e93-0b67-40fe-a946-d8fc39a126c6 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:47.973166503 +0000 UTC m=+34.233479007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh8l9" (UniqueName: "kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9") pod "network-check-target-fnq9v" (UID: "7f0a7e93-0b67-40fe-a946-d8fc39a126c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:32.123799 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:32.123695 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:32.123799 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:32.123742 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:32.123799 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:32.123695 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:32.124023 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:32.123835 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:32.124023 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:32.123958 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:32.124109 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:32.124028 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:34.126721 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:34.126687 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:34.127113 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:34.126816 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:34.127113 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:34.126878 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:34.127113 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:34.126972 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:34.127380 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:34.127363 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:34.127479 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:34.127460 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:36.123865 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:36.123836 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:36.124261 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:36.123835 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:36.124261 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:36.123936 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:36.124261 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:36.123841 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:36.124261 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:36.124015 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:36.124261 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:36.124121 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:37.250931 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.250285 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2w9bg" event={"ID":"fa158ca8-9798-4ad3-b302-a69eb51e1295","Type":"ContainerStarted","Data":"c7f41ca6a58c25e4e1c8c3795014d56b05e118be6087ac3ec12dff234479f290"} Jul 30 20:21:37.252300 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.252266 2537 generic.go:356] "Generic (PLEG): container finished" podID="3ce791b2-1232-4fbe-9ffe-523b7263a619" containerID="f3f2dea36381b7106cc418430fb73d0f8a198b886e26924a2b5bd0855eed3d42" exitCode=0 Jul 30 20:21:37.252429 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.252348 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerDied","Data":"f3f2dea36381b7106cc418430fb73d0f8a198b886e26924a2b5bd0855eed3d42"} Jul 30 20:21:37.254417 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.254358 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" event={"ID":"88b6c1c4-8292-4249-81ff-7c1e860c75f7","Type":"ContainerStarted","Data":"f86e43a49f2bf0ca2c515130b012201e917e5e2cf834db398a1876c01d5de2b5"} Jul 30 20:21:37.255872 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.255850 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-97pmk" event={"ID":"0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9","Type":"ContainerStarted","Data":"7855ba4f19950c4f7adba304e2257c0b8568c641a33a2320b9a375b8b5a0885a"} Jul 30 20:21:37.259904 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.259880 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"5990e2f983e4fd9b02799fbf36ce174e65070bb232dbc9578a5e3c2f031a6249"} Jul 30 20:21:37.260166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.259955 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"72ed6fa20ee324ea9759660bee0ca22a31647e9eb4d18aaf9ca76b7fd41e5df2"} Jul 30 20:21:37.260166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.260005 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"4e15a6a26cb95e11d0b6713c00f5c3643896776ec6c64547d8b69f35b0c860e4"} Jul 30 20:21:37.260166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.260056 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"64061c2d30dca859045bd72686d662e45a9e8d60a2496391105b39c5dc34bc26"} Jul 30 20:21:37.260166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.260106 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"31f412da4ef16d99f119db43c44474a2d4f94855c398fc330e5d111537cd5046"} Jul 30 20:21:37.260166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.260150 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"0f34d7bd711079bf0ad2fa02fbe4083f9e37338c91377150ae78bce00616e15f"} Jul 30 20:21:37.261438 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.261414 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b9lwj" event={"ID":"16f37bef-efea-4c62-b4ca-fcb69b4ecaad","Type":"ContainerStarted","Data":"36ae444b26667cfadba237b71702a364de1b8109246223698702777477f8f509"} Jul 30 20:21:37.263107 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.263082 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s7jf2" event={"ID":"67bcbfeb-4460-4bc4-bc79-0cfcc04f0936","Type":"ContainerStarted","Data":"948f43491a926e27a47bfda0b6c044060ff9b969ff6e12ee6b85406b17a9c0b3"} Jul 30 20:21:37.264704 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.264683 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" event={"ID":"94f79733-5c93-41b4-8960-4ffd7f4f0592","Type":"ContainerStarted","Data":"81db81d6762d5b8a0da046afcf546d708c35f8323db0687fe96f551bbdbdb23f"} Jul 30 20:21:37.268741 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.268695 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2w9bg" podStartSLOduration=4.055573229 podStartE2EDuration="23.268681931s" totalImagesPullingTime="19.213108702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:37.268633755 +0000 UTC m=+23.528946280" watchObservedRunningTime="2026-07-30 20:21:37.268681931 +0000 UTC m=+23.528994456" Jul 30 20:21:37.269125 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.269091 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-223.ec2.internal" podStartSLOduration=22.269077709 podStartE2EDuration="22.269077709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:19.227927793 +0000 UTC m=+5.488240322" watchObservedRunningTime="2026-07-30 20:21:37.269077709 +0000 UTC m=+23.529390238" Jul 30 20:21:37.304740 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.304676 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s7jf2" podStartSLOduration=4.079509001 podStartE2EDuration="23.304654662s" totalImagesPullingTime="19.225145661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:37.304078231 +0000 UTC m=+23.564390757" watchObservedRunningTime="2026-07-30 20:21:37.304654662 +0000 UTC m=+23.564967185" Jul 30 20:21:37.305056 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.305026 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b9lwj" podStartSLOduration=4.098681144 podStartE2EDuration="23.305015784s" totalImagesPullingTime="19.20633464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:37.284834782 +0000 UTC m=+23.545147309" watchObservedRunningTime="2026-07-30 20:21:37.305015784 +0000 UTC m=+23.565328312" Jul 30 20:21:37.326819 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.326746 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-97pmk" podStartSLOduration=4.109122912 podStartE2EDuration="23.326723524s" totalImagesPullingTime="19.217600612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:37.325560841 +0000 UTC m=+23.585873368" watchObservedRunningTime="2026-07-30 20:21:37.326723524 +0000 UTC m=+23.587036096" Jul 30 20:21:37.349378 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.349323 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b7qhp" podStartSLOduration=4.141027089 podStartE2EDuration="23.349301242s" totalImagesPullingTime="19.208274153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:37.348934168 +0000 UTC m=+23.609246695" watchObservedRunningTime="2026-07-30 20:21:37.349301242 +0000 UTC m=+23.609613767" Jul 30 20:21:37.396052 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.396014 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:37.512801 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.512697 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:37.614326 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:37.614294 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:38.024348 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.024300 2537 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 20:21:38.121346 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.121158 2537 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T20:21:38.024326351Z","UUID":"b787c699-9f26-4219-8997-c5ad73096f58","Handler":null,"Name":"","Endpoint":""} Jul 30 20:21:38.122900 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.122875 2537 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 20:21:38.123012 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.122920 2537 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 20:21:38.123757 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.123736 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:38.123757 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.123751 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:38.123981 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:38.123875 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:38.123981 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:38.123954 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:38.124082 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.124028 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:38.124142 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:38.124122 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:38.269452 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.269410 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ttb9" event={"ID":"680f84f3-d507-47cf-bc45-157fb05484f4","Type":"ContainerStarted","Data":"f3515e04a92e762bfc832aed77bc3c0522c116429f79005bb2aba3b9c5987a05"} Jul 30 20:21:38.271514 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.271479 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" event={"ID":"88b6c1c4-8292-4249-81ff-7c1e860c75f7","Type":"ContainerStarted","Data":"1651af35fcedbc216a07940dc236866ecafd66185ac5a69396d860c30e48e6bb"} Jul 30 20:21:38.288460 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.288399 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9ttb9" podStartSLOduration=5.050760094 podStartE2EDuration="24.288379151s" totalImagesPullingTime="19.237619057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:38.288049815 +0000 UTC m=+24.548362342" watchObservedRunningTime="2026-07-30 20:21:38.288379151 +0000 UTC m=+24.548691677" Jul 30 20:21:38.407767 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.407671 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:38.509634 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:38.509600 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:39.050140 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:39.050101 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:39.382677 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:39.381916 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:39.382677 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:39.382544 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:40.123594 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.123554 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:40.123823 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.123583 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:40.123823 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.123554 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:40.123823 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:40.123686 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:40.123823 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:40.123792 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:40.124034 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:40.123884 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:40.282733 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.281190 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"c7340ef483de2bb7eaab64adb44b3dbb17339bf313de3b09c94fc97805edaf5e"} Jul 30 20:21:40.282733 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.281327 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:40.282733 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.281771 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-97pmk" Jul 30 20:21:40.359895 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.359856 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:40.445159 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:40.445075 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:41.380930 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:41.380895 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:42.124556 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.124385 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:42.125253 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.124385 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:42.125253 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:42.124631 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:42.125253 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:42.124716 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:42.125253 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.124387 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:42.125253 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:42.124811 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:42.247168 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.247135 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:42.287585 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.287496 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" event={"ID":"88b6c1c4-8292-4249-81ff-7c1e860c75f7","Type":"ContainerStarted","Data":"d3148a2b7079ba0fe8b36d8c0840ca37d9102fa0e1015492aaaa6083f5bcbfba"} Jul 30 20:21:42.291069 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.291043 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" event={"ID":"fc85afc9-0be3-43a2-b2b0-96a4143daa28","Type":"ContainerStarted","Data":"8f33054e18aa8fcd17d3a102c92077e4c2efd42de0c056cdca01dacb26554e1b"} Jul 30 20:21:42.291287 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.291262 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:42.292899 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.292876 2537 generic.go:356] "Generic (PLEG): container finished" podID="3ce791b2-1232-4fbe-9ffe-523b7263a619" containerID="1f1a57adc6ae79e4176f9414482d838e43dc5553a0b7604b964f3b896d90d6ae" exitCode=0 Jul 30 20:21:42.292978 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.292946 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerDied","Data":"1f1a57adc6ae79e4176f9414482d838e43dc5553a0b7604b964f3b896d90d6ae"} Jul 30 20:21:42.301602 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.301574 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:42.304809 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.304762 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7qzgs" podStartSLOduration=4.272831295 podStartE2EDuration="28.304747635s" totalImagesPullingTime="24.03191634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:42.304465855 +0000 UTC m=+28.564778381" watchObservedRunningTime="2026-07-30 20:21:42.304747635 +0000 UTC m=+28.565060160" Jul 30 20:21:42.307688 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.307670 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:42.329056 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:42.329009 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" podStartSLOduration=9.002823588 podStartE2EDuration="28.328994394s" totalImagesPullingTime="19.326170806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:42.328816443 +0000 UTC m=+28.589128968" watchObservedRunningTime="2026-07-30 20:21:42.328994394 +0000 UTC m=+28.589306919" Jul 30 20:21:43.296372 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.296343 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:43.296771 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.296460 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:43.311209 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.311177 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:21:43.764847 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.764670 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gr4lp"] Jul 30 20:21:43.765012 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.764944 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:43.765084 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:43.765050 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:43.768428 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.768401 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fnq9v"] Jul 30 20:21:43.768571 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.768530 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:43.768649 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:43.768628 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:43.769643 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.769622 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2k9d"] Jul 30 20:21:43.769740 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:43.769728 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:43.769872 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:43.769847 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:44.299601 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:44.299563 2537 generic.go:356] "Generic (PLEG): container finished" podID="3ce791b2-1232-4fbe-9ffe-523b7263a619" containerID="b8ddf384a449a8866b3bc171856cf807bcb0096b7f0794115fae29c0c5f5a845" exitCode=0 Jul 30 20:21:44.300139 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:44.299650 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerDied","Data":"b8ddf384a449a8866b3bc171856cf807bcb0096b7f0794115fae29c0c5f5a845"} Jul 30 20:21:45.123898 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:45.123863 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:45.124082 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:45.123863 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:45.124082 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:45.123999 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:45.124082 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:45.124038 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:46.124316 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.124204 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:46.125087 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:46.124334 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:46.259544 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.259515 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:46.305984 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.305950 2537 generic.go:356] "Generic (PLEG): container finished" podID="3ce791b2-1232-4fbe-9ffe-523b7263a619" containerID="e4679cbb507f14416d30db9ad47552a04e224da604f5d399cce80ec4ed78bad3" exitCode=0 Jul 30 20:21:46.306143 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.306011 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerDied","Data":"e4679cbb507f14416d30db9ad47552a04e224da604f5d399cce80ec4ed78bad3"} Jul 30 20:21:46.333497 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.333465 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:46.419744 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.419668 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:46.523716 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.523687 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:46.646487 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.646448 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:46.728179 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.728081 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:46.815543 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:46.815505 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.104805 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.104694 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.124336 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.124308 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:47.124801 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.124423 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr4lp" podUID="0e4765f1-4ab8-4d6c-8f93-0ee04514feb1" Jul 30 20:21:47.124801 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.124470 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:47.124801 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.124600 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q2k9d" podUID="0de7db11-2876-47d9-bbd3-c2a8faf8bf8d" Jul 30 20:21:47.244295 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.244254 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.344516 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.344482 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.447508 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.447397 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.507939 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.507902 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.788230 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.788145 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:47.788358 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.788308 2537 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:47.788422 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.788409 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret podName:0e4765f1-4ab8-4d6c-8f93-0ee04514feb1 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:19.788387508 +0000 UTC m=+66.048700025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret") pod "global-pull-secret-syncer-gr4lp" (UID: "0e4765f1-4ab8-4d6c-8f93-0ee04514feb1") : object "kube-system"/"original-pull-secret" not registered Jul 30 20:21:47.854717 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.854612 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.888609 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.888574 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:47.888867 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.888711 2537 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:47.888867 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.888807 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs podName:0de7db11-2876-47d9-bbd3-c2a8faf8bf8d nodeName:}" failed. No retries permitted until 2026-07-30 20:22:19.88876241 +0000 UTC m=+66.149074914 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs") pod "network-metrics-daemon-q2k9d" (UID: "0de7db11-2876-47d9-bbd3-c2a8faf8bf8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 20:21:47.941500 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.941457 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:47.989643 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:47.989608 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:47.989849 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.989813 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 20:21:47.989849 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.989834 2537 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 20:21:47.989849 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.989846 2537 projected.go:196] Error preparing data for projected volume kube-api-access-mh8l9 for pod openshift-network-diagnostics/network-check-target-fnq9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:47.990003 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:47.989919 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9 podName:7f0a7e93-0b67-40fe-a946-d8fc39a126c6 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:19.98990042 +0000 UTC m=+66.250212926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh8l9" (UniqueName: "kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9") pod "network-check-target-fnq9v" (UID: "7f0a7e93-0b67-40fe-a946-d8fc39a126c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 20:21:48.123540 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:48.123506 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:48.123711 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:48.123631 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnq9v" podUID="7f0a7e93-0b67-40fe-a946-d8fc39a126c6" Jul 30 20:21:48.142089 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:48.141999 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:48.217684 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:48.217646 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:49.038654 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.038462 2537 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-223.ec2.internal" event="NodeReady" Jul 30 20:21:49.038828 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.038758 2537 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 20:21:49.084149 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.084122 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s99zs"] Jul 30 20:21:49.089273 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.089247 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.091107 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.091076 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f2jbj"] Jul 30 20:21:49.091669 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.091640 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7hfpr\"" Jul 30 20:21:49.092378 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.092358 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 20:21:49.092486 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.092455 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 20:21:49.092546 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.092455 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 20:21:49.095719 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.095700 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.096728 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.096709 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s99zs"] Jul 30 20:21:49.098266 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.098131 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9q9km\"" Jul 30 20:21:49.098357 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.098339 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 20:21:49.098571 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.098556 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 20:21:49.098571 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.098564 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 20:21:49.098731 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.098600 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 20:21:49.104642 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.104178 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sphlb"] Jul 30 20:21:49.112004 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.111979 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2jbj"] Jul 30 20:21:49.112113 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.112099 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.115102 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.115083 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 20:21:49.115249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.115228 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 20:21:49.115335 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.115304 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nmld7\"" Jul 30 20:21:49.116900 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.116882 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sphlb"] Jul 30 20:21:49.124419 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.124397 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:21:49.124575 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.124555 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:21:49.127199 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.127181 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4qz5l\"" Jul 30 20:21:49.127199 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.127192 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 20:21:49.127354 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.127247 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 20:21:49.198868 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.198828 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.198868 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.198878 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hqw\" (UniqueName: \"kubernetes.io/projected/1eb47f98-77c5-46a2-a3af-edbebbfb9d82-kube-api-access-z5hqw\") pod \"ingress-canary-s99zs\" (UID: \"1eb47f98-77c5-46a2-a3af-edbebbfb9d82\") " pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.198913 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f0009-1ced-4345-aad3-618ab9cf8546-config-volume\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.198966 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab8496e-e598-4ff6-b83c-c35c6aeb6320-crio-socket\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.199005 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab8496e-e598-4ff6-b83c-c35c6aeb6320-data-volume\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.199035 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45k7\" (UniqueName: \"kubernetes.io/projected/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-api-access-d45k7\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.199107 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.199144 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.199177 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1f0009-1ced-4345-aad3-618ab9cf8546-tmp-dir\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.199220 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eb47f98-77c5-46a2-a3af-edbebbfb9d82-cert\") pod \"ingress-canary-s99zs\" (UID: \"1eb47f98-77c5-46a2-a3af-edbebbfb9d82\") " pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.199355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.199249 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvp4\" (UniqueName: \"kubernetes.io/projected/ea1f0009-1ced-4345-aad3-618ab9cf8546-kube-api-access-wvvp4\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.238475 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.238377 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2w9bg_fa158ca8-9798-4ad3-b302-a69eb51e1295/dns-node-resolver/0.log" Jul 30 20:21:49.299837 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.299790 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.299837 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.299838 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hqw\" (UniqueName: \"kubernetes.io/projected/1eb47f98-77c5-46a2-a3af-edbebbfb9d82-kube-api-access-z5hqw\") pod \"ingress-canary-s99zs\" (UID: \"1eb47f98-77c5-46a2-a3af-edbebbfb9d82\") " pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.300045 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.299894 2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 20:21:49.300045 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.299980 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:49.799964465 +0000 UTC m=+36.060276972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : configmap "kube-rbac-proxy" not found Jul 30 20:21:49.300159 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300123 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f0009-1ced-4345-aad3-618ab9cf8546-config-volume\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.300216 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300160 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab8496e-e598-4ff6-b83c-c35c6aeb6320-crio-socket\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.300216 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300182 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab8496e-e598-4ff6-b83c-c35c6aeb6320-data-volume\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.300216 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300208 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45k7\" (UniqueName: \"kubernetes.io/projected/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-api-access-d45k7\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.300340 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300246 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.300389 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300361 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.300442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300393 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aab8496e-e598-4ff6-b83c-c35c6aeb6320-crio-socket\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.300481 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.300438 2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:49.300481 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.300445 2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:49.300558 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.300496 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:49.800479384 +0000 UTC m=+36.060791896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:49.300558 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.300515 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls podName:ea1f0009-1ced-4345-aad3-618ab9cf8546 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:49.800505262 +0000 UTC m=+36.060817779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls") pod "dns-default-sphlb" (UID: "ea1f0009-1ced-4345-aad3-618ab9cf8546") : secret "dns-default-metrics-tls" not found Jul 30 20:21:49.300558 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300532 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1f0009-1ced-4345-aad3-618ab9cf8546-tmp-dir\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.300715 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300560 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eb47f98-77c5-46a2-a3af-edbebbfb9d82-cert\") pod \"ingress-canary-s99zs\" (UID: \"1eb47f98-77c5-46a2-a3af-edbebbfb9d82\") " pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.300715 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300594 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvp4\" (UniqueName: \"kubernetes.io/projected/ea1f0009-1ced-4345-aad3-618ab9cf8546-kube-api-access-wvvp4\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.300715 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300595 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aab8496e-e598-4ff6-b83c-c35c6aeb6320-data-volume\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.300890 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300709 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1f0009-1ced-4345-aad3-618ab9cf8546-config-volume\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.300890 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.300881 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1f0009-1ced-4345-aad3-618ab9cf8546-tmp-dir\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.305621 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.305592 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1eb47f98-77c5-46a2-a3af-edbebbfb9d82-cert\") pod \"ingress-canary-s99zs\" (UID: \"1eb47f98-77c5-46a2-a3af-edbebbfb9d82\") " pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.308911 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.308887 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45k7\" (UniqueName: \"kubernetes.io/projected/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-api-access-d45k7\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.309262 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.309243 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvp4\" (UniqueName: \"kubernetes.io/projected/ea1f0009-1ced-4345-aad3-618ab9cf8546-kube-api-access-wvvp4\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.309523 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.309503 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hqw\" (UniqueName: \"kubernetes.io/projected/1eb47f98-77c5-46a2-a3af-edbebbfb9d82-kube-api-access-z5hqw\") pod \"ingress-canary-s99zs\" (UID: \"1eb47f98-77c5-46a2-a3af-edbebbfb9d82\") " pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.402530 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.402486 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s99zs" Jul 30 20:21:49.721030 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.720998 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-96vfc"] Jul 30 20:21:49.723946 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.723921 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.726424 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.726400 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 30 20:21:49.726764 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.726747 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 30 20:21:49.727521 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.727503 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 30 20:21:49.727615 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.727503 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9b9g9\"" Jul 30 20:21:49.727615 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.727553 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 30 20:21:49.733824 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.733800 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-96vfc"] Jul 30 20:21:49.798954 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.798916 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:49.805981 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.805956 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.806140 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.806003 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pghp\" (UniqueName: \"kubernetes.io/projected/fd79c52d-15cb-4237-b5b6-5d2dd9636232-kube-api-access-5pghp\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.806140 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.806033 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:49.806140 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.806063 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:49.806140 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.806079 2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 20:21:49.806140 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.806106 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fd79c52d-15cb-4237-b5b6-5d2dd9636232-signing-key\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.806140 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.806130 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fd79c52d-15cb-4237-b5b6-5d2dd9636232-signing-cabundle\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.806350 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.806146 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:50.806127131 +0000 UTC m=+37.066439680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : configmap "kube-rbac-proxy" not found Jul 30 20:21:49.806350 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.806151 2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:49.806350 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.806185 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls podName:ea1f0009-1ced-4345-aad3-618ab9cf8546 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:50.806174438 +0000 UTC m=+37.066486954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls") pod "dns-default-sphlb" (UID: "ea1f0009-1ced-4345-aad3-618ab9cf8546") : secret "dns-default-metrics-tls" not found Jul 30 20:21:49.806350 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.806242 2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:49.806350 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:49.806301 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:50.806284671 +0000 UTC m=+37.066597177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:49.858960 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.858924 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:49.906946 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.906902 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fd79c52d-15cb-4237-b5b6-5d2dd9636232-signing-key\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.907129 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.906952 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fd79c52d-15cb-4237-b5b6-5d2dd9636232-signing-cabundle\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.907129 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.907030 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pghp\" (UniqueName: \"kubernetes.io/projected/fd79c52d-15cb-4237-b5b6-5d2dd9636232-kube-api-access-5pghp\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.907827 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.907804 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fd79c52d-15cb-4237-b5b6-5d2dd9636232-signing-cabundle\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.909726 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.909702 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fd79c52d-15cb-4237-b5b6-5d2dd9636232-signing-key\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:49.918121 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:49.918098 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pghp\" (UniqueName: \"kubernetes.io/projected/fd79c52d-15cb-4237-b5b6-5d2dd9636232-kube-api-access-5pghp\") pod \"service-ca-98c88fbb5-96vfc\" (UID: \"fd79c52d-15cb-4237-b5b6-5d2dd9636232\") " pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:50.035038 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.034955 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" Jul 30 20:21:50.123886 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.123853 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:21:50.126822 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.126800 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 20:21:50.126960 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.126847 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jh8wl\"" Jul 30 20:21:50.126960 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.126860 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 20:21:50.239058 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.239032 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b9lwj_16f37bef-efea-4c62-b4ca-fcb69b4ecaad/node-ca/0.log" Jul 30 20:21:50.443352 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.443324 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-96vfc"] Jul 30 20:21:50.446532 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:50.446504 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd79c52d_15cb_4237_b5b6_5d2dd9636232.slice/crio-e6a86f4d2bcd01b9df48007869d64753613c0cd2ef2a6e29e54c9116341cdacb WatchSource:0}: Error finding container e6a86f4d2bcd01b9df48007869d64753613c0cd2ef2a6e29e54c9116341cdacb: Status 404 returned error can't find the container with id e6a86f4d2bcd01b9df48007869d64753613c0cd2ef2a6e29e54c9116341cdacb Jul 30 20:21:50.451764 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.451740 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s99zs"] Jul 30 20:21:50.466113 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:21:50.466064 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb47f98_77c5_46a2_a3af_edbebbfb9d82.slice/crio-19fab656361713decbadd751f7432c33bc1662ecadfe3913cad7eb75a3fa949b WatchSource:0}: Error finding container 19fab656361713decbadd751f7432c33bc1662ecadfe3913cad7eb75a3fa949b: Status 404 returned error can't find the container with id 19fab656361713decbadd751f7432c33bc1662ecadfe3913cad7eb75a3fa949b Jul 30 20:21:50.767261 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.767085 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:50.815076 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.814988 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:50.815076 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.815040 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:50.815076 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.815068 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:50.815300 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:50.815113 2537 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 20:21:50.815300 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:50.815182 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.815163936 +0000 UTC m=+39.075476458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : configmap "kube-rbac-proxy" not found Jul 30 20:21:50.815300 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:50.815200 2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:50.815300 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:50.815206 2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:50.815300 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:50.815234 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.815223741 +0000 UTC m=+39.075536245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:50.815300 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:50.815260 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls podName:ea1f0009-1ced-4345-aad3-618ab9cf8546 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:52.815244899 +0000 UTC m=+39.075557406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls") pod "dns-default-sphlb" (UID: "ea1f0009-1ced-4345-aad3-618ab9cf8546") : secret "dns-default-metrics-tls" not found Jul 30 20:21:50.886622 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.886589 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:50.988748 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:50.988712 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:51.322453 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:51.322375 2537 generic.go:356] "Generic (PLEG): container finished" podID="3ce791b2-1232-4fbe-9ffe-523b7263a619" containerID="7175aafd725c41f28837fe616943e2445ee3be954fea5edc124945abdbeaddb4" exitCode=0 Jul 30 20:21:51.322946 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:51.322490 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerDied","Data":"7175aafd725c41f28837fe616943e2445ee3be954fea5edc124945abdbeaddb4"} Jul 30 20:21:51.323866 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:51.323678 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s99zs" event={"ID":"1eb47f98-77c5-46a2-a3af-edbebbfb9d82","Type":"ContainerStarted","Data":"19fab656361713decbadd751f7432c33bc1662ecadfe3913cad7eb75a3fa949b"} Jul 30 20:21:51.325359 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:51.325337 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" event={"ID":"fd79c52d-15cb-4237-b5b6-5d2dd9636232","Type":"ContainerStarted","Data":"e6a86f4d2bcd01b9df48007869d64753613c0cd2ef2a6e29e54c9116341cdacb"} Jul 30 20:21:52.330264 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:52.330231 2537 generic.go:356] "Generic (PLEG): container finished" podID="3ce791b2-1232-4fbe-9ffe-523b7263a619" containerID="c3c814a25bd846fd65e20738a9e7b4c2d6b34c8cb4bb355360f773bdd0dea677" exitCode=0 Jul 30 20:21:52.330851 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:52.330292 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerDied","Data":"c3c814a25bd846fd65e20738a9e7b4c2d6b34c8cb4bb355360f773bdd0dea677"} Jul 30 20:21:52.830249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:52.830210 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:52.830443 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:52.830270 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:52.830443 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:52.830297 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:52.830443 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:52.830429 2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:52.830600 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:52.830496 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:56.830477276 +0000 UTC m=+43.090789782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:52.830600 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:52.830429 2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:52.830600 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:52.830569 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls podName:ea1f0009-1ced-4345-aad3-618ab9cf8546 nodeName:}" failed. No retries permitted until 2026-07-30 20:21:56.830552647 +0000 UTC m=+43.090865150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls") pod "dns-default-sphlb" (UID: "ea1f0009-1ced-4345-aad3-618ab9cf8546") : secret "dns-default-metrics-tls" not found Jul 30 20:21:52.830877 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:52.830850 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aab8496e-e598-4ff6-b83c-c35c6aeb6320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:52.860459 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:52.860419 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:53.250873 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:53.250836 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:53.540203 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:53.540168 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:53.640884 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:53.640843 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:54.338327 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:54.338282 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s99zs" event={"ID":"1eb47f98-77c5-46a2-a3af-edbebbfb9d82","Type":"ContainerStarted","Data":"c2b9ab506275695bbfa4f03660f1719b3d0226698b7fbdf2a59f763408ad0003"} Jul 30 20:21:54.339753 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:54.339727 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" event={"ID":"fd79c52d-15cb-4237-b5b6-5d2dd9636232","Type":"ContainerStarted","Data":"456e92872216622257664111e6a8a8b87ce86c3b50aaa72bb63110cd747c398c"} Jul 30 20:21:54.342941 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:54.342920 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k5h49" event={"ID":"3ce791b2-1232-4fbe-9ffe-523b7263a619","Type":"ContainerStarted","Data":"3a81c525f9d7a63b254b6602d0ab1ee8cc49a4adcf5f8a5e4ad4a6dd2bc250b7"} Jul 30 20:21:54.355820 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:54.355761 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s99zs" podStartSLOduration=2.382426701 podStartE2EDuration="5.355744001s" totalImagesPullingTime="2.9733173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:54.35534071 +0000 UTC m=+40.615653235" watchObservedRunningTime="2026-07-30 20:21:54.355744001 +0000 UTC m=+40.616056529" Jul 30 20:21:54.379469 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:54.379403 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k5h49" podStartSLOduration=10.733334401 podStartE2EDuration="40.379383463s" totalImagesPullingTime="29.646049062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:54.37847317 +0000 UTC m=+40.638785697" watchObservedRunningTime="2026-07-30 20:21:54.379383463 +0000 UTC m=+40.639695992" Jul 30 20:21:54.394134 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:54.394079 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-96vfc" podStartSLOduration=2.402141356 podStartE2EDuration="5.394065461s" totalImagesPullingTime="2.991924105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:21:54.393452567 +0000 UTC m=+40.653765094" watchObservedRunningTime="2026-07-30 20:21:54.394065461 +0000 UTC m=+40.654377986" Jul 30 20:21:54.908765 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:54.908725 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:55.064769 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:55.064731 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:55.166142 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:55.166040 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:55.267035 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:55.267002 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:56.010949 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:56.010910 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:56.411697 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:56.411655 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:56.814562 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:56.814461 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:56.861447 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:56.861414 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:21:56.861635 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:56.861457 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:21:56.861635 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:56.861565 2537 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 20:21:56.861635 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:56.861611 2537 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 20:21:56.861635 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:56.861634 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls podName:ea1f0009-1ced-4345-aad3-618ab9cf8546 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:04.861617067 +0000 UTC m=+51.121929573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls") pod "dns-default-sphlb" (UID: "ea1f0009-1ced-4345-aad3-618ab9cf8546") : secret "dns-default-metrics-tls" not found Jul 30 20:21:56.861866 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:21:56.861658 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls podName:aab8496e-e598-4ff6-b83c-c35c6aeb6320 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:04.861642326 +0000 UTC m=+51.121954831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2jbj" (UID: "aab8496e-e598-4ff6-b83c-c35c6aeb6320") : secret "insights-runtime-extractor-tls" not found Jul 30 20:21:56.942326 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:56.942289 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:57.042643 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:57.042607 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:57.603273 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:57.603241 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:58.004107 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:58.004066 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:58.214593 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:58.214552 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:58.603487 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:58.603436 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:58.664911 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:58.664881 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:59.054151 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:59.054116 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:59.403051 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:59.403020 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:21:59.804431 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:21:59.804314 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:00.295640 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:00.295599 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:00.396661 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:00.396622 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:00.608132 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:00.608101 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:01.256788 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:01.256739 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:01.411122 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:01.411075 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:01.605912 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:01.605830 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:02.004313 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:02.004277 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:02.218431 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:02.218393 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:02.811423 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:02.811386 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:03.217485 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:03.217448 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:03.621246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:03.621216 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:03.814017 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:03.813976 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:04.215560 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:04.215528 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:04.593935 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:04.593847 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:04.927198 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:04.927153 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:22:04.927198 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:04.927201 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:22:04.930393 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:04.930365 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea1f0009-1ced-4345-aad3-618ab9cf8546-metrics-tls\") pod \"dns-default-sphlb\" (UID: \"ea1f0009-1ced-4345-aad3-618ab9cf8546\") " pod="openshift-dns/dns-default-sphlb" Jul 30 20:22:04.930499 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:04.930442 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aab8496e-e598-4ff6-b83c-c35c6aeb6320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2jbj\" (UID: \"aab8496e-e598-4ff6-b83c-c35c6aeb6320\") " pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:22:04.958740 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:04.958688 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:05.011226 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.011186 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2jbj" Jul 30 20:22:05.022363 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.022330 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sphlb" Jul 30 20:22:05.159379 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.159297 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2jbj"] Jul 30 20:22:05.182060 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.181267 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sphlb"] Jul 30 20:22:05.184312 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:05.184284 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab8496e_e598_4ff6_b83c_c35c6aeb6320.slice/crio-c6fe73223c081d6849d1c2f40019b17b33dbd4d7551490ddc73462a8d4230338 WatchSource:0}: Error finding container c6fe73223c081d6849d1c2f40019b17b33dbd4d7551490ddc73462a8d4230338: Status 404 returned error can't find the container with id c6fe73223c081d6849d1c2f40019b17b33dbd4d7551490ddc73462a8d4230338 Jul 30 20:22:05.185525 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:05.185496 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1f0009_1ced_4345_aad3_618ab9cf8546.slice/crio-27d04898adf8fd1d03e1719293d2aa116639ab40e5ef13debe5a18117b7f17ba WatchSource:0}: Error finding container 27d04898adf8fd1d03e1719293d2aa116639ab40e5ef13debe5a18117b7f17ba: Status 404 returned error can't find the container with id 27d04898adf8fd1d03e1719293d2aa116639ab40e5ef13debe5a18117b7f17ba Jul 30 20:22:05.203938 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.203904 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:05.381355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.381325 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2jbj" event={"ID":"aab8496e-e598-4ff6-b83c-c35c6aeb6320","Type":"ContainerStarted","Data":"0b0cad282a0f7ce7c94e0c5613905f2a91caeeb269a02e5f1fda334aab475528"} Jul 30 20:22:05.381524 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.381376 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2jbj" event={"ID":"aab8496e-e598-4ff6-b83c-c35c6aeb6320","Type":"ContainerStarted","Data":"c6fe73223c081d6849d1c2f40019b17b33dbd4d7551490ddc73462a8d4230338"} Jul 30 20:22:05.382366 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.382344 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sphlb" event={"ID":"ea1f0009-1ced-4345-aad3-618ab9cf8546","Type":"ContainerStarted","Data":"27d04898adf8fd1d03e1719293d2aa116639ab40e5ef13debe5a18117b7f17ba"} Jul 30 20:22:05.404894 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.404865 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:05.602172 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:05.602093 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:07.388187 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:07.388107 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2jbj" event={"ID":"aab8496e-e598-4ff6-b83c-c35c6aeb6320","Type":"ContainerStarted","Data":"0086572cf79dd0c7d586ab51e4caedceb19510fba8b986e98c95a5f8fc846ecf"} Jul 30 20:22:08.397213 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:08.397175 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sphlb" event={"ID":"ea1f0009-1ced-4345-aad3-618ab9cf8546","Type":"ContainerStarted","Data":"722e68fcaf8f3a4ac0e51fc65cb64269361d302afcae3a6b5ab43b9f6982078c"} Jul 30 20:22:08.397690 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:08.397238 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sphlb" event={"ID":"ea1f0009-1ced-4345-aad3-618ab9cf8546","Type":"ContainerStarted","Data":"8d516eaf14392f3dcb881acb01ce2d18448b6d3861ba2fb2c705b76db7a7be57"} Jul 30 20:22:08.397690 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:08.397342 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sphlb" Jul 30 20:22:08.419380 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:08.419333 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sphlb" podStartSLOduration=17.1397802 podStartE2EDuration="19.419317915s" totalImagesPullingTime="2.279537715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:08.419109983 +0000 UTC m=+54.679422509" watchObservedRunningTime="2026-07-30 20:22:08.419317915 +0000 UTC m=+54.679630419" Jul 30 20:22:09.033483 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.033450 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f"] Jul 30 20:22:09.036338 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.036316 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.037540 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.037514 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld"] Jul 30 20:22:09.039426 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.039411 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vtc4g\"" Jul 30 20:22:09.040189 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.040167 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 30 20:22:09.040263 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.040198 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 30 20:22:09.040698 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.040679 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 30 20:22:09.040921 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.040720 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 30 20:22:09.040969 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.040954 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.046136 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.046117 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 30 20:22:09.046319 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.046305 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 30 20:22:09.048714 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.048699 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 30 20:22:09.051190 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.051176 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 30 20:22:09.056232 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.056213 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f"] Jul 30 20:22:09.058030 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.057961 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-ca\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.058030 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.058006 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxqc\" (UniqueName: \"kubernetes.io/projected/09c9ff5b-add7-45cc-a265-c1cdd24030ed-kube-api-access-ndxqc\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.058185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.058048 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-hub\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.058185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.058063 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xsd5\" (UniqueName: \"kubernetes.io/projected/d9898637-30be-4f52-8004-dd697bc0a9e2-kube-api-access-9xsd5\") pod \"managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f\" (UID: \"d9898637-30be-4f52-8004-dd697bc0a9e2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.058185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.058094 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c9ff5b-add7-45cc-a265-c1cdd24030ed-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.058185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.058143 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.058185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.058163 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.058185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.058183 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9898637-30be-4f52-8004-dd697bc0a9e2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f\" (UID: \"d9898637-30be-4f52-8004-dd697bc0a9e2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.071557 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.071529 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld"] Jul 30 20:22:09.128547 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.128512 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:09.154018 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.153989 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-777fbbd874-dkkms"] Jul 30 20:22:09.156890 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.156874 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.158474 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158452 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.158581 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158477 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9898637-30be-4f52-8004-dd697bc0a9e2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f\" (UID: \"d9898637-30be-4f52-8004-dd697bc0a9e2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.158581 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158523 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-ca\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.158581 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158543 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxqc\" (UniqueName: \"kubernetes.io/projected/09c9ff5b-add7-45cc-a265-c1cdd24030ed-kube-api-access-ndxqc\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.158674 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158636 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-hub\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.158674 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158664 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xsd5\" (UniqueName: \"kubernetes.io/projected/d9898637-30be-4f52-8004-dd697bc0a9e2-kube-api-access-9xsd5\") pod \"managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f\" (UID: \"d9898637-30be-4f52-8004-dd697bc0a9e2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.158759 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158694 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c9ff5b-add7-45cc-a265-c1cdd24030ed-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.158759 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.158724 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.160460 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.160428 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c9ff5b-add7-45cc-a265-c1cdd24030ed-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.161936 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.161853 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 20:22:09.162336 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.162194 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 20:22:09.162486 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.162373 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.162832 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.162708 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.164008 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.163447 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 20:22:09.164008 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.163913 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-hub\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.164189 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.164049 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q4l4z\"" Jul 30 20:22:09.164401 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.164233 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/09c9ff5b-add7-45cc-a265-c1cdd24030ed-ca\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.165029 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.165013 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9898637-30be-4f52-8004-dd697bc0a9e2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f\" (UID: \"d9898637-30be-4f52-8004-dd697bc0a9e2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.173647 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.173618 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 20:22:09.179693 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.179672 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxqc\" (UniqueName: \"kubernetes.io/projected/09c9ff5b-add7-45cc-a265-c1cdd24030ed-kube-api-access-ndxqc\") pod \"cluster-proxy-proxy-agent-6fcbb849bf-g6jld\" (UID: \"09c9ff5b-add7-45cc-a265-c1cdd24030ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.189303 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.189273 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xsd5\" (UniqueName: \"kubernetes.io/projected/d9898637-30be-4f52-8004-dd697bc0a9e2-kube-api-access-9xsd5\") pod \"managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f\" (UID: \"d9898637-30be-4f52-8004-dd697bc0a9e2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.191213 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.191187 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777fbbd874-dkkms"] Jul 30 20:22:09.236076 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.236040 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:09.259811 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.259762 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-registry-tls\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.259974 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.259873 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csg45\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-kube-api-access-csg45\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.259974 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.259896 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-bound-sa-token\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.259974 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.259916 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5f1cc0-1374-4d34-9c52-654943b71645-installation-pull-secrets\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.260110 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.259993 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5f1cc0-1374-4d34-9c52-654943b71645-image-registry-private-configuration\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.260110 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.260017 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5f1cc0-1374-4d34-9c52-654943b71645-ca-trust-extracted\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.260110 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.260032 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f1cc0-1374-4d34-9c52-654943b71645-trusted-ca\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.260110 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.260057 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5f1cc0-1374-4d34-9c52-654943b71645-registry-certificates\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.334641 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.334535 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:09.358902 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.358871 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" Jul 30 20:22:09.361165 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361143 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5f1cc0-1374-4d34-9c52-654943b71645-image-registry-private-configuration\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361266 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361174 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5f1cc0-1374-4d34-9c52-654943b71645-ca-trust-extracted\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361266 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361190 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f1cc0-1374-4d34-9c52-654943b71645-trusted-ca\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361266 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361213 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5f1cc0-1374-4d34-9c52-654943b71645-registry-certificates\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361396 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361336 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-registry-tls\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361464 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361445 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csg45\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-kube-api-access-csg45\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361523 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361480 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-bound-sa-token\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361523 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361509 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5f1cc0-1374-4d34-9c52-654943b71645-installation-pull-secrets\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.361618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.361541 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5f1cc0-1374-4d34-9c52-654943b71645-ca-trust-extracted\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.362166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.362142 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5f1cc0-1374-4d34-9c52-654943b71645-registry-certificates\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.362630 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.362606 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f1cc0-1374-4d34-9c52-654943b71645-trusted-ca\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.363898 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.363876 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5f1cc0-1374-4d34-9c52-654943b71645-installation-pull-secrets\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.364039 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.364018 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-registry-tls\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.364099 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.364050 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5f1cc0-1374-4d34-9c52-654943b71645-image-registry-private-configuration\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.364740 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.364722 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" Jul 30 20:22:09.379848 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.379755 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-bound-sa-token\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.379848 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.379764 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csg45\" (UniqueName: \"kubernetes.io/projected/ee5f1cc0-1374-4d34-9c52-654943b71645-kube-api-access-csg45\") pod \"image-registry-777fbbd874-dkkms\" (UID: \"ee5f1cc0-1374-4d34-9c52-654943b71645\") " pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.406423 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.406363 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2jbj" event={"ID":"aab8496e-e598-4ff6-b83c-c35c6aeb6320","Type":"ContainerStarted","Data":"b2ce76089670fdfc3f1b2b81e38a0b2850a04eabd76526c28f62905138d60300"} Jul 30 20:22:09.427211 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.426702 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f2jbj" podStartSLOduration=17.296844406 podStartE2EDuration="20.426677607s" totalImagesPullingTime="3.129833201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:09.425564584 +0000 UTC m=+55.685877125" watchObservedRunningTime="2026-07-30 20:22:09.426677607 +0000 UTC m=+55.686990134" Jul 30 20:22:09.431006 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.430930 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:09.481094 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.481046 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:09.501271 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.501240 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f"] Jul 30 20:22:09.504553 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:09.504524 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9898637_30be_4f52_8004_dd697bc0a9e2.slice/crio-cb8050c231e46a24f1a7203d172fc6c89120b954c52825e4585e71c1f8ccf303 WatchSource:0}: Error finding container cb8050c231e46a24f1a7203d172fc6c89120b954c52825e4585e71c1f8ccf303: Status 404 returned error can't find the container with id cb8050c231e46a24f1a7203d172fc6c89120b954c52825e4585e71c1f8ccf303 Jul 30 20:22:09.518177 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.518109 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld"] Jul 30 20:22:09.522244 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:09.522216 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09c9ff5b_add7_45cc_a265_c1cdd24030ed.slice/crio-f63d26086a62b3e4b749f7f2e8cf89d48d1bef183de51e951bfe1fbaa2c46dfb WatchSource:0}: Error finding container f63d26086a62b3e4b749f7f2e8cf89d48d1bef183de51e951bfe1fbaa2c46dfb: Status 404 returned error can't find the container with id f63d26086a62b3e4b749f7f2e8cf89d48d1bef183de51e951bfe1fbaa2c46dfb Jul 30 20:22:09.610565 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:09.610537 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777fbbd874-dkkms"] Jul 30 20:22:09.613560 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:09.613538 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5f1cc0_1374_4d34_9c52_654943b71645.slice/crio-d1611cbb16f385495b6752137787c2ef9a87883b4695b31b5016b88048af53ce WatchSource:0}: Error finding container d1611cbb16f385495b6752137787c2ef9a87883b4695b31b5016b88048af53ce: Status 404 returned error can't find the container with id d1611cbb16f385495b6752137787c2ef9a87883b4695b31b5016b88048af53ce Jul 30 20:22:10.414563 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:10.413692 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777fbbd874-dkkms" event={"ID":"ee5f1cc0-1374-4d34-9c52-654943b71645","Type":"ContainerStarted","Data":"8865bb40f2013b002b1d9c8ac3733026765998eaa817b279013c2a62eebaec02"} Jul 30 20:22:10.414563 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:10.413759 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777fbbd874-dkkms" event={"ID":"ee5f1cc0-1374-4d34-9c52-654943b71645","Type":"ContainerStarted","Data":"d1611cbb16f385495b6752137787c2ef9a87883b4695b31b5016b88048af53ce"} Jul 30 20:22:10.414563 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:10.414510 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:10.417255 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:10.416988 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" event={"ID":"09c9ff5b-add7-45cc-a265-c1cdd24030ed","Type":"ContainerStarted","Data":"f63d26086a62b3e4b749f7f2e8cf89d48d1bef183de51e951bfe1fbaa2c46dfb"} Jul 30 20:22:10.420471 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:10.420403 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" event={"ID":"d9898637-30be-4f52-8004-dd697bc0a9e2","Type":"ContainerStarted","Data":"cb8050c231e46a24f1a7203d172fc6c89120b954c52825e4585e71c1f8ccf303"} Jul 30 20:22:10.589628 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:10.589519 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:10.653166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:10.653117 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:13.184134 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:13.184095 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:13.248019 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:13.247979 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:15.135309 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.135269 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:15.241054 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.241009 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:15.306224 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.306192 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:15.318940 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.318918 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdnvq" Jul 30 20:22:15.344587 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.344513 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-777fbbd874-dkkms" podStartSLOduration=6.344485125 podStartE2EDuration="6.344485125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:10.435602435 +0000 UTC m=+56.695914963" watchObservedRunningTime="2026-07-30 20:22:15.344485125 +0000 UTC m=+61.604797639" Jul 30 20:22:15.438613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.438527 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" event={"ID":"09c9ff5b-add7-45cc-a265-c1cdd24030ed","Type":"ContainerStarted","Data":"8ea234e0b183742adb652dbea7985fc938dbebd86bcffdbd41b93825e325d204"} Jul 30 20:22:15.438613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.438577 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" event={"ID":"09c9ff5b-add7-45cc-a265-c1cdd24030ed","Type":"ContainerStarted","Data":"78660ced82a66e07eb55c82835e17aadc6a3c48e15b98c893cd5e06e74f56bf3"} Jul 30 20:22:15.438613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.438596 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" event={"ID":"09c9ff5b-add7-45cc-a265-c1cdd24030ed","Type":"ContainerStarted","Data":"ff0c1f748dc39e7eab5bf266101461cd3a7df635943dc26f9f2255fe2bacf2b0"} Jul 30 20:22:15.439898 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.439877 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" event={"ID":"d9898637-30be-4f52-8004-dd697bc0a9e2","Type":"ContainerStarted","Data":"a312ee79eb9ad4f91b796c6dee7a4d99bcf97da0661ccd91520a9bdeef09945b"} Jul 30 20:22:15.452173 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.452132 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:15.462859 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.462817 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcbb849bf-g6jld" podStartSLOduration=1.395433133 podStartE2EDuration="6.462801428s" totalImagesPullingTime="5.067368295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:15.462201522 +0000 UTC m=+61.722514075" watchObservedRunningTime="2026-07-30 20:22:15.462801428 +0000 UTC m=+61.723113946" Jul 30 20:22:15.477947 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.477898 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74dd6c8fbb-7xq8f" podStartSLOduration=1.410033613 podStartE2EDuration="6.477882057s" totalImagesPullingTime="5.067848444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:15.476933442 +0000 UTC m=+61.737245975" watchObservedRunningTime="2026-07-30 20:22:15.477882057 +0000 UTC m=+61.738194610" Jul 30 20:22:15.523041 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:15.523006 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:17.320887 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:17.320848 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:17.382219 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:17.382169 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:17.628553 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:17.628523 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:18.409422 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:18.409391 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sphlb" Jul 30 20:22:19.210406 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.210365 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:19.269066 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.269033 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:19.844420 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.844383 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:22:19.847012 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.846994 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 20:22:19.857344 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.857314 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e4765f1-4ab8-4d6c-8f93-0ee04514feb1-original-pull-secret\") pod \"global-pull-secret-syncer-gr4lp\" (UID: \"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1\") " pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:22:19.945146 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.945104 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:22:19.947644 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.947624 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 20:22:19.957957 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:19.957933 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0de7db11-2876-47d9-bbd3-c2a8faf8bf8d-metrics-certs\") pod \"network-metrics-daemon-q2k9d\" (UID: \"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d\") " pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:22:20.036706 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.036669 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr4lp" Jul 30 20:22:20.045118 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.045093 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4qz5l\"" Jul 30 20:22:20.045507 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.045478 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:22:20.047823 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.047611 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 20:22:20.053511 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.053494 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q2k9d" Jul 30 20:22:20.057906 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.057885 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 20:22:20.068894 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.068872 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8l9\" (UniqueName: \"kubernetes.io/projected/7f0a7e93-0b67-40fe-a946-d8fc39a126c6-kube-api-access-mh8l9\") pod \"network-check-target-fnq9v\" (UID: \"7f0a7e93-0b67-40fe-a946-d8fc39a126c6\") " pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:22:20.136234 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.136016 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jh8wl\"" Jul 30 20:22:20.144416 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.144393 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:22:20.185515 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.185436 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gr4lp"] Jul 30 20:22:20.188554 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:20.188529 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4765f1_4ab8_4d6c_8f93_0ee04514feb1.slice/crio-081a72e46ebf1b1cbcc37d5d6291ad3de3f9b2a7053035b7c1241b25b988f1a2 WatchSource:0}: Error finding container 081a72e46ebf1b1cbcc37d5d6291ad3de3f9b2a7053035b7c1241b25b988f1a2: Status 404 returned error can't find the container with id 081a72e46ebf1b1cbcc37d5d6291ad3de3f9b2a7053035b7c1241b25b988f1a2 Jul 30 20:22:20.203003 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.202930 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q2k9d"] Jul 30 20:22:20.206399 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:20.206370 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de7db11_2876_47d9_bbd3_c2a8faf8bf8d.slice/crio-5b9186409b43f3bb7d8351d90723297980251503f3df1e49b8476303553e44c5 WatchSource:0}: Error finding container 5b9186409b43f3bb7d8351d90723297980251503f3df1e49b8476303553e44c5: Status 404 returned error can't find the container with id 5b9186409b43f3bb7d8351d90723297980251503f3df1e49b8476303553e44c5 Jul 30 20:22:20.264727 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.264696 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:20.291029 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.291002 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fnq9v"] Jul 30 20:22:20.294265 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:20.294240 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0a7e93_0b67_40fe_a946_d8fc39a126c6.slice/crio-bd12158b517a8ecb93e53553bb7f96c09f7cfca0af5de56bb428bf0769498ee8 WatchSource:0}: Error finding container bd12158b517a8ecb93e53553bb7f96c09f7cfca0af5de56bb428bf0769498ee8: Status 404 returned error can't find the container with id bd12158b517a8ecb93e53553bb7f96c09f7cfca0af5de56bb428bf0769498ee8 Jul 30 20:22:20.372822 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.372793 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:20.448846 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.448803 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:20.457873 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.457838 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gr4lp" event={"ID":"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1","Type":"ContainerStarted","Data":"081a72e46ebf1b1cbcc37d5d6291ad3de3f9b2a7053035b7c1241b25b988f1a2"} Jul 30 20:22:20.458931 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.458906 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fnq9v" event={"ID":"7f0a7e93-0b67-40fe-a946-d8fc39a126c6","Type":"ContainerStarted","Data":"bd12158b517a8ecb93e53553bb7f96c09f7cfca0af5de56bb428bf0769498ee8"} Jul 30 20:22:20.459985 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.459964 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2k9d" event={"ID":"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d","Type":"ContainerStarted","Data":"5b9186409b43f3bb7d8351d90723297980251503f3df1e49b8476303553e44c5"} Jul 30 20:22:20.507559 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.507519 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:20.692047 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.691954 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:20.866806 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:20.866646 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:21.066279 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:21.066096 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:21.165624 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:21.165573 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:21.468534 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:21.468492 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:21.569609 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:21.569532 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:22.082065 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:22.081985 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:22.352598 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:22.352507 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:22.470998 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:22.470957 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2k9d" event={"ID":"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d","Type":"ContainerStarted","Data":"a2ccdb51ca04035becc0365b8ef31bbb804af391f59bfb034e83b91334841ca4"} Jul 30 20:22:22.471158 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:22.471023 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q2k9d" event={"ID":"0de7db11-2876-47d9-bbd3-c2a8faf8bf8d","Type":"ContainerStarted","Data":"56cef661c98a9e283d74712e9d7d1805c6ace5046ea749283b1cf2faffacfbba"} Jul 30 20:22:22.490771 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:22.490714 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q2k9d" podStartSLOduration=67.164563085 podStartE2EDuration="1m8.490699303s" totalImagesPullingTime="1.326136218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:22.490177001 +0000 UTC m=+68.750489529" watchObservedRunningTime="2026-07-30 20:22:22.490699303 +0000 UTC m=+68.751011837" Jul 30 20:22:22.585082 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:22.585044 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:22.684081 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:22.684040 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:23.324613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:23.324556 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:23.387989 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:23.387948 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:23.612210 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:23.612157 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:24.337356 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:24.337297 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:24.437911 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:24.437864 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:24.495828 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:24.495791 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:25.222709 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.222673 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:25.466279 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.466246 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:25.484564 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.484476 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fnq9v" event={"ID":"7f0a7e93-0b67-40fe-a946-d8fc39a126c6","Type":"ContainerStarted","Data":"efac514f377db3be05f4e1b0095f4c5a90aa5f10815bc78d040ab025d3fc9f94"} Jul 30 20:22:25.484702 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.484574 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:22:25.486023 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.485992 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gr4lp" event={"ID":"0e4765f1-4ab8-4d6c-8f93-0ee04514feb1","Type":"ContainerStarted","Data":"08b0eddbd0c9d8dafe21532a9cf4d52cb3746808fa9b1b2df6d46d8eaa142287"} Jul 30 20:22:25.500818 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.500750 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fnq9v" podStartSLOduration=66.951458086 podStartE2EDuration="1m11.500736771s" totalImagesPullingTime="4.549278685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:25.4999063 +0000 UTC m=+71.760218827" watchObservedRunningTime="2026-07-30 20:22:25.500736771 +0000 UTC m=+71.761049297" Jul 30 20:22:25.515273 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.515219 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gr4lp" podStartSLOduration=66.855766258 podStartE2EDuration="1m11.515200924s" totalImagesPullingTime="4.659434666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:25.514919134 +0000 UTC m=+71.775231662" watchObservedRunningTime="2026-07-30 20:22:25.515200924 +0000 UTC m=+71.775513451" Jul 30 20:22:25.634171 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.634136 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:25.749263 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.749171 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hxhwk"] Jul 30 20:22:25.751829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.751809 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.754867 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.754844 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 20:22:25.755003 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.754889 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 20:22:25.755303 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.755260 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 20:22:25.755303 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.755295 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 20:22:25.755545 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.755526 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lm9q\"" Jul 30 20:22:25.756059 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.756040 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 20:22:25.756295 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.756281 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 20:22:25.759661 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.759603 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:25.792386 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792348 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-sys\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792577 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792394 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-tls\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792577 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792490 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792577 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792529 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclzh\" (UniqueName: \"kubernetes.io/projected/443e5a17-c7d3-405e-b103-fdee927b1a99-kube-api-access-xclzh\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792702 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792596 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-root\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792702 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792643 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792702 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792679 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-textfile\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792820 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792716 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/443e5a17-c7d3-405e-b103-fdee927b1a99-metrics-client-ca\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.792820 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.792742 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-wtmp\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.858308 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.858270 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:25.893442 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893400 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-tls\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893466 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893492 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclzh\" (UniqueName: \"kubernetes.io/projected/443e5a17-c7d3-405e-b103-fdee927b1a99-kube-api-access-xclzh\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893516 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-root\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893548 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:22:25.893574 2537 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893625 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-root\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:22:25.893650 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-tls podName:443e5a17-c7d3-405e-b103-fdee927b1a99 nodeName:}" failed. No retries permitted until 2026-07-30 20:22:26.393626612 +0000 UTC m=+72.653939121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-tls") pod "node-exporter-hxhwk" (UID: "443e5a17-c7d3-405e-b103-fdee927b1a99") : secret "node-exporter-tls" not found Jul 30 20:22:25.893671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893578 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-textfile\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893688 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/443e5a17-c7d3-405e-b103-fdee927b1a99-metrics-client-ca\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893714 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-wtmp\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893757 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-sys\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893847 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-sys\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893962 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-textfile\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.893968 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-wtmp\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894456 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.894165 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-accelerators-collector-config\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.894456 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.894293 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/443e5a17-c7d3-405e-b103-fdee927b1a99-metrics-client-ca\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.896070 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.896048 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.903087 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.903067 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclzh\" (UniqueName: \"kubernetes.io/projected/443e5a17-c7d3-405e-b103-fdee927b1a99-kube-api-access-xclzh\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:25.962569 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:25.962533 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:26.057634 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.057535 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:26.158834 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.158806 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:26.395965 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.395932 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:26.398844 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.398824 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-tls\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:26.401075 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.401055 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/443e5a17-c7d3-405e-b103-fdee927b1a99-node-exporter-tls\") pod \"node-exporter-hxhwk\" (UID: \"443e5a17-c7d3-405e-b103-fdee927b1a99\") " pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:26.663006 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.662927 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hxhwk" Jul 30 20:22:26.674917 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:22:26.674886 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443e5a17_c7d3_405e_b103_fdee927b1a99.slice/crio-27fecbc0dedb2f277aed2202718048234697396c261cddeb2fc7472559144a04 WatchSource:0}: Error finding container 27fecbc0dedb2f277aed2202718048234697396c261cddeb2fc7472559144a04: Status 404 returned error can't find the container with id 27fecbc0dedb2f277aed2202718048234697396c261cddeb2fc7472559144a04 Jul 30 20:22:26.708888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.708856 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:26.804145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.804099 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:26.917688 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:26.917596 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:27.493619 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:27.493586 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxhwk" event={"ID":"443e5a17-c7d3-405e-b103-fdee927b1a99","Type":"ContainerStarted","Data":"27fecbc0dedb2f277aed2202718048234697396c261cddeb2fc7472559144a04"} Jul 30 20:22:28.498916 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:28.498880 2537 generic.go:356] "Generic (PLEG): container finished" podID="443e5a17-c7d3-405e-b103-fdee927b1a99" containerID="5f3c58107c984da0dc071881083cdb8aa37ca1e472d878bd2c0617a819a2131c" exitCode=0 Jul 30 20:22:28.499307 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:28.498970 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxhwk" event={"ID":"443e5a17-c7d3-405e-b103-fdee927b1a99","Type":"ContainerDied","Data":"5f3c58107c984da0dc071881083cdb8aa37ca1e472d878bd2c0617a819a2131c"} Jul 30 20:22:28.694919 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:28.694883 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:28.767155 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:28.767063 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:28.993312 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:28.993266 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:29.507018 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:29.506988 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxhwk" event={"ID":"443e5a17-c7d3-405e-b103-fdee927b1a99","Type":"ContainerStarted","Data":"3c2292ac11d72b3a38299b68b6169821405b5944f8e3ff5f6c619260b7346140"} Jul 30 20:22:29.507415 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:29.507037 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hxhwk" event={"ID":"443e5a17-c7d3-405e-b103-fdee927b1a99","Type":"ContainerStarted","Data":"6f5c45d5af5332eef26d1403833dfec755c1304ec7f2c5dd328475a76e8437ad"} Jul 30 20:22:29.526440 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:29.526363 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hxhwk" podStartSLOduration=3.627355609 podStartE2EDuration="4.52634582s" totalImagesPullingTime="837.780667ms" totalInitContainerRuntime="61.209544ms" isStatefulPod=true podCreationTimestamp="2026-07-30 20:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:22:29.526028892 +0000 UTC m=+75.786341418" watchObservedRunningTime="2026-07-30 20:22:29.52634582 +0000 UTC m=+75.786658401" Jul 30 20:22:30.495684 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:30.495648 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:30.588953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:30.588915 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:30.689891 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:30.689856 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:32.027711 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:32.027666 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:32.431384 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:32.431355 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-777fbbd874-dkkms" Jul 30 20:22:32.557630 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:32.557591 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:32.627195 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:32.627165 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:32.727809 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:32.727702 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:32.788045 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:32.788013 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:34.238905 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:34.238870 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:34.301047 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:34.301017 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:38.099110 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:38.098983 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:38.199480 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:38.199438 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:38.250835 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:38.250798 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:40.110578 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:40.110545 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:40.165091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:40.165050 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:43.312221 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:43.312188 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:43.413064 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:43.413030 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:46.300943 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:46.300907 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:46.396067 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:46.396028 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:47.277860 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:47.277826 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:47.332544 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:47.332504 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:53.711587 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:53.711546 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:53.762699 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:53.762659 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:55.103958 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:55.103914 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:55.204682 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:55.204628 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:56.491539 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:56.491507 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fnq9v" Jul 30 20:22:59.507344 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:59.507314 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:22:59.607769 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:22:59.607730 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:04.215502 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:04.215464 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:04.315519 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:04.315484 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:09.201014 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:09.200977 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:21.583250 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:21.583213 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:41.843012 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:41.842974 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:41.898630 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:41.898594 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:53.743836 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:53.743771 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:53.795305 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:53.795263 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:58.045544 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:58.045493 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:58.103803 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:58.103751 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:58.197281 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:58.197243 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:58.253529 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:58.253491 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:58.785430 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:58.785382 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:23:58.835297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:23:58.835262 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:04.216219 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:04.216182 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:04.314357 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:04.314323 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:09.096100 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:09.095838 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:09.197790 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:09.197721 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:09.256800 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:09.256749 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:35.552222 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:35.552175 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:35.649925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:35.649891 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:47.956071 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:47.956034 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:48.056734 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:48.056694 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:24:49.586587 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:24:49.586547 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:02.157051 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:02.157012 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:02.961028 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:02.960988 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:05.763608 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:05.763577 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:05.863236 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:05.863200 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:07.611850 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:07.611817 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:07.748798 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:07.748749 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:09.966546 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:09.966512 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:10.385919 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:10.385882 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:12.567212 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:12.567177 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:12.625218 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:12.625184 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:15.341394 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:15.341346 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:15.397117 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:15.397075 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:17.383527 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:17.383483 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:18.970407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:18.970371 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:19.068828 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:19.068796 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:21.213766 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.213735 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68"] Jul 30 20:25:21.216739 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.216717 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.219067 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.219041 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 30 20:25:21.219199 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.219103 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gz5t8\"" Jul 30 20:25:21.219868 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.219851 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 30 20:25:21.226616 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.226593 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68"] Jul 30 20:25:21.324121 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.324063 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclvh\" (UniqueName: \"kubernetes.io/projected/d4d19509-3783-44dd-9d0a-9d4d46a63320-kube-api-access-sclvh\") pod \"openshift-lws-operator-fd8ccff4c-j5q68\" (UID: \"d4d19509-3783-44dd-9d0a-9d4d46a63320\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.324310 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.324160 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d19509-3783-44dd-9d0a-9d4d46a63320-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j5q68\" (UID: \"d4d19509-3783-44dd-9d0a-9d4d46a63320\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.424639 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.424605 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d19509-3783-44dd-9d0a-9d4d46a63320-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j5q68\" (UID: \"d4d19509-3783-44dd-9d0a-9d4d46a63320\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.424836 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.424662 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclvh\" (UniqueName: \"kubernetes.io/projected/d4d19509-3783-44dd-9d0a-9d4d46a63320-kube-api-access-sclvh\") pod \"openshift-lws-operator-fd8ccff4c-j5q68\" (UID: \"d4d19509-3783-44dd-9d0a-9d4d46a63320\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.425008 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.424989 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4d19509-3783-44dd-9d0a-9d4d46a63320-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j5q68\" (UID: \"d4d19509-3783-44dd-9d0a-9d4d46a63320\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.435535 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.435504 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclvh\" (UniqueName: \"kubernetes.io/projected/d4d19509-3783-44dd-9d0a-9d4d46a63320-kube-api-access-sclvh\") pod \"openshift-lws-operator-fd8ccff4c-j5q68\" (UID: \"d4d19509-3783-44dd-9d0a-9d4d46a63320\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.526215 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.526117 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" Jul 30 20:25:21.656301 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:21.656268 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68"] Jul 30 20:25:21.659247 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:25:21.659218 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4d19509_3783_44dd_9d0a_9d4d46a63320.slice/crio-52a44b616ca6763cc9f438075e5abcc3c0d173dd6765fa4fcc0d1db375a28fa6 WatchSource:0}: Error finding container 52a44b616ca6763cc9f438075e5abcc3c0d173dd6765fa4fcc0d1db375a28fa6: Status 404 returned error can't find the container with id 52a44b616ca6763cc9f438075e5abcc3c0d173dd6765fa4fcc0d1db375a28fa6 Jul 30 20:25:22.049684 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:22.049651 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" event={"ID":"d4d19509-3783-44dd-9d0a-9d4d46a63320","Type":"ContainerStarted","Data":"52a44b616ca6763cc9f438075e5abcc3c0d173dd6765fa4fcc0d1db375a28fa6"} Jul 30 20:25:25.062616 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:25.062580 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" event={"ID":"d4d19509-3783-44dd-9d0a-9d4d46a63320","Type":"ContainerStarted","Data":"89e9db35e0ca5923c0b6aa9cbccb6db83bb9dc1afda47dc17ce08ea181800938"} Jul 30 20:25:25.077690 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:25.077499 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5q68" podStartSLOduration=1.408469173 podStartE2EDuration="4.077479927s" totalImagesPullingTime="2.669010754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:25.07728164 +0000 UTC m=+251.337594147" watchObservedRunningTime="2026-07-30 20:25:25.077479927 +0000 UTC m=+251.337792454" Jul 30 20:25:25.856120 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:25.856091 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:25.956538 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:25.956507 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:27.545607 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:27.545574 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:27.756466 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:27.756434 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:27.857451 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:27.857426 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:37.755304 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:37.755223 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:37.854809 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:37.854740 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:40.777196 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:40.777157 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:40.878405 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:40.878369 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:41.586060 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.586008 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:41.845949 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.845871 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:41.941646 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.941613 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:41.981598 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.979706 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd"] Jul 30 20:25:41.985601 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.985574 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:41.988355 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.988330 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 20:25:41.988462 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.988334 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wcr5p\"" Jul 30 20:25:41.988462 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.988396 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 20:25:41.988665 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.988649 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 20:25:41.988732 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.988716 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 20:25:41.997997 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:41.997976 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd"] Jul 30 20:25:42.067077 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.067048 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-tmp\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.067231 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.067082 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bbn\" (UniqueName: \"kubernetes.io/projected/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-kube-api-access-l2bbn\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.067231 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.067103 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.067231 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.067128 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.067231 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.067198 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-manifests\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.109558 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.109527 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:42.168007 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.167976 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-manifests\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.168155 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.168028 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-tmp\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.168155 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.168059 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bbn\" (UniqueName: \"kubernetes.io/projected/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-kube-api-access-l2bbn\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.168155 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.168082 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.168155 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.168120 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.168428 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.168331 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-manifests\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.170634 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.170606 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-tmp\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.171008 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.170988 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.171060 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.170988 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.175577 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.175556 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bbn\" (UniqueName: \"kubernetes.io/projected/85f353d8-4ba4-4d60-929e-686dcb6c3cc4-kube-api-access-l2bbn\") pod \"opendatahub-operator-controller-manager-58fb4f68b-f5dfd\" (UID: \"85f353d8-4ba4-4d60-929e-686dcb6c3cc4\") " pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.211144 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.211117 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:42.295680 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.295658 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:42.311035 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.310998 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:42.424356 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.424321 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:42.436695 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.436670 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd"] Jul 30 20:25:42.440397 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:25:42.440376 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f353d8_4ba4_4d60_929e_686dcb6c3cc4.slice/crio-bab8689cbf24bb3c3b4335641376d437b77192d41d4d451d12be22ad267f37cc WatchSource:0}: Error finding container bab8689cbf24bb3c3b4335641376d437b77192d41d4d451d12be22ad267f37cc: Status 404 returned error can't find the container with id bab8689cbf24bb3c3b4335641376d437b77192d41d4d451d12be22ad267f37cc Jul 30 20:25:42.530938 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:42.530906 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:43.122465 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:43.122432 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" event={"ID":"85f353d8-4ba4-4d60-929e-686dcb6c3cc4","Type":"ContainerStarted","Data":"bab8689cbf24bb3c3b4335641376d437b77192d41d4d451d12be22ad267f37cc"} Jul 30 20:25:46.136464 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:46.136423 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" event={"ID":"85f353d8-4ba4-4d60-929e-686dcb6c3cc4","Type":"ContainerStarted","Data":"62769292a350199843a12592f72a79cd90e22e1ecbb4403c3af12569f908c2b7"} Jul 30 20:25:47.142137 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:47.142103 2537 generic.go:356] "Generic (PLEG): container finished" podID="85f353d8-4ba4-4d60-929e-686dcb6c3cc4" containerID="62769292a350199843a12592f72a79cd90e22e1ecbb4403c3af12569f908c2b7" exitCode=0 Jul 30 20:25:47.142505 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:47.142183 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" event={"ID":"85f353d8-4ba4-4d60-929e-686dcb6c3cc4","Type":"ContainerDied","Data":"62769292a350199843a12592f72a79cd90e22e1ecbb4403c3af12569f908c2b7"} Jul 30 20:25:48.147281 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:48.147192 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" event={"ID":"85f353d8-4ba4-4d60-929e-686dcb6c3cc4","Type":"ContainerStarted","Data":"1b5cc6bc81572ce09a5192a6077f00d33bb6ed6433186b2f506ec797397f03bb"} Jul 30 20:25:48.147705 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:48.147305 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:25:48.167790 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:48.167743 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" podStartSLOduration=2.346235177 podStartE2EDuration="7.167729525s" totalImagesPullingTime="3.817495266s" totalInitContainerRuntime="1.003999082s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:25:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:25:48.16673316 +0000 UTC m=+274.427045690" watchObservedRunningTime="2026-07-30 20:25:48.167729525 +0000 UTC m=+274.428042043" Jul 30 20:25:53.194075 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:53.194027 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:53.299632 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:53.299594 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:25:59.153173 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:25:59.153122 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58fb4f68b-f5dfd" Jul 30 20:26:00.933266 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:00.933220 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:01.087485 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.087449 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:01.283560 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.283466 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:01.312918 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.312893 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn"] Jul 30 20:26:01.321116 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.321097 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.324586 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.324565 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 20:26:01.324706 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.324669 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 30 20:26:01.324791 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.324763 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 30 20:26:01.324857 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.324811 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 20:26:01.325609 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.325593 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cq7sc\"" Jul 30 20:26:01.345700 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.345676 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn"] Jul 30 20:26:01.407749 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.407716 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-tls-certs\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.407933 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.407756 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgsg\" (UniqueName: \"kubernetes.io/projected/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-kube-api-access-5fgsg\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.407933 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.407797 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-tmp\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.508740 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.508702 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-tls-certs\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.508922 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.508749 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgsg\" (UniqueName: \"kubernetes.io/projected/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-kube-api-access-5fgsg\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.508922 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.508802 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-tmp\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.511058 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.511040 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-tmp\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.511291 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.511274 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-tls-certs\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.526283 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.526257 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgsg\" (UniqueName: \"kubernetes.io/projected/f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9-kube-api-access-5fgsg\") pod \"kube-auth-proxy-55c4ff7d78-ch2mn\" (UID: \"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9\") " pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.545718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.545651 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:01.630410 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.630381 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" Jul 30 20:26:01.764843 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.764813 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn"] Jul 30 20:26:01.768363 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:26:01.768341 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8584ac5_cd46_4ee1_b320_fbe5e6dbb7a9.slice/crio-882f6b7e9f53970d517efdaa0b794b51ab8cf6c838b6c93768b0017bd10170a1 WatchSource:0}: Error finding container 882f6b7e9f53970d517efdaa0b794b51ab8cf6c838b6c93768b0017bd10170a1: Status 404 returned error can't find the container with id 882f6b7e9f53970d517efdaa0b794b51ab8cf6c838b6c93768b0017bd10170a1 Jul 30 20:26:01.859756 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.859729 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2m8b5"] Jul 30 20:26:01.864132 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.864113 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:01.867175 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.867153 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4wnnv\"" Jul 30 20:26:01.867175 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.867168 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 30 20:26:01.888300 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.888278 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2m8b5"] Jul 30 20:26:01.911976 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.911954 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782tj\" (UniqueName: \"kubernetes.io/projected/c5913a92-4b24-439d-bcbf-810625bf88d0-kube-api-access-782tj\") pod \"odh-model-controller-7998c67dc4-2m8b5\" (UID: \"c5913a92-4b24-439d-bcbf-810625bf88d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:01.912066 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:01.911984 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5913a92-4b24-439d-bcbf-810625bf88d0-cert\") pod \"odh-model-controller-7998c67dc4-2m8b5\" (UID: \"c5913a92-4b24-439d-bcbf-810625bf88d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:02.013146 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.013116 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782tj\" (UniqueName: \"kubernetes.io/projected/c5913a92-4b24-439d-bcbf-810625bf88d0-kube-api-access-782tj\") pod \"odh-model-controller-7998c67dc4-2m8b5\" (UID: \"c5913a92-4b24-439d-bcbf-810625bf88d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:02.013146 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.013151 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5913a92-4b24-439d-bcbf-810625bf88d0-cert\") pod \"odh-model-controller-7998c67dc4-2m8b5\" (UID: \"c5913a92-4b24-439d-bcbf-810625bf88d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:02.013603 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:26:02.013244 2537 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 30 20:26:02.013603 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:26:02.013300 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5913a92-4b24-439d-bcbf-810625bf88d0-cert podName:c5913a92-4b24-439d-bcbf-810625bf88d0 nodeName:}" failed. No retries permitted until 2026-07-30 20:26:02.513285595 +0000 UTC m=+288.773598099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c5913a92-4b24-439d-bcbf-810625bf88d0-cert") pod "odh-model-controller-7998c67dc4-2m8b5" (UID: "c5913a92-4b24-439d-bcbf-810625bf88d0") : secret "odh-model-controller-webhook-cert" not found Jul 30 20:26:02.044043 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.044016 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:02.045371 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.045350 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-782tj\" (UniqueName: \"kubernetes.io/projected/c5913a92-4b24-439d-bcbf-810625bf88d0-kube-api-access-782tj\") pod \"odh-model-controller-7998c67dc4-2m8b5\" (UID: \"c5913a92-4b24-439d-bcbf-810625bf88d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:02.140649 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.140564 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:02.195708 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.195675 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" event={"ID":"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9","Type":"ContainerStarted","Data":"882f6b7e9f53970d517efdaa0b794b51ab8cf6c838b6c93768b0017bd10170a1"} Jul 30 20:26:02.248806 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.248764 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:02.492452 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.492366 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:02.518631 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.518585 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5913a92-4b24-439d-bcbf-810625bf88d0-cert\") pod \"odh-model-controller-7998c67dc4-2m8b5\" (UID: \"c5913a92-4b24-439d-bcbf-810625bf88d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:02.521226 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.521202 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5913a92-4b24-439d-bcbf-810625bf88d0-cert\") pod \"odh-model-controller-7998c67dc4-2m8b5\" (UID: \"c5913a92-4b24-439d-bcbf-810625bf88d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:02.773548 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.773472 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:02.935004 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:02.934831 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2m8b5"] Jul 30 20:26:03.009806 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.009720 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jc9qc"] Jul 30 20:26:03.014237 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.014220 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.017498 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.017477 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 30 20:26:03.033880 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.033564 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jc9qc"] Jul 30 20:26:03.034185 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.034065 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:03.123955 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.123913 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d994d8-e020-4b34-aa88-586ada797d49-tls-certs\") pod \"model-serving-api-54d89bd5fc-jc9qc\" (UID: \"69d994d8-e020-4b34-aa88-586ada797d49\") " pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.124145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.124038 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xlv\" (UniqueName: \"kubernetes.io/projected/69d994d8-e020-4b34-aa88-586ada797d49-kube-api-access-j8xlv\") pod \"model-serving-api-54d89bd5fc-jc9qc\" (UID: \"69d994d8-e020-4b34-aa88-586ada797d49\") " pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.199941 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.199908 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" event={"ID":"c5913a92-4b24-439d-bcbf-810625bf88d0","Type":"ContainerStarted","Data":"92b02882266672869012d60c2d589e916f6de2d31aec83b0c1e6270149c49919"} Jul 30 20:26:03.225257 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.225217 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d994d8-e020-4b34-aa88-586ada797d49-tls-certs\") pod \"model-serving-api-54d89bd5fc-jc9qc\" (UID: \"69d994d8-e020-4b34-aa88-586ada797d49\") " pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.225400 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.225295 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xlv\" (UniqueName: \"kubernetes.io/projected/69d994d8-e020-4b34-aa88-586ada797d49-kube-api-access-j8xlv\") pod \"model-serving-api-54d89bd5fc-jc9qc\" (UID: \"69d994d8-e020-4b34-aa88-586ada797d49\") " pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.225400 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:26:03.225354 2537 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 30 20:26:03.225519 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:26:03.225434 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d994d8-e020-4b34-aa88-586ada797d49-tls-certs podName:69d994d8-e020-4b34-aa88-586ada797d49 nodeName:}" failed. No retries permitted until 2026-07-30 20:26:03.725412721 +0000 UTC m=+289.985725236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/69d994d8-e020-4b34-aa88-586ada797d49-tls-certs") pod "model-serving-api-54d89bd5fc-jc9qc" (UID: "69d994d8-e020-4b34-aa88-586ada797d49") : secret "model-serving-api-tls" not found Jul 30 20:26:03.226198 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.226170 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:03.231241 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.231220 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl"] Jul 30 20:26:03.234585 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.234568 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.236305 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.236283 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xlv\" (UniqueName: \"kubernetes.io/projected/69d994d8-e020-4b34-aa88-586ada797d49-kube-api-access-j8xlv\") pod \"model-serving-api-54d89bd5fc-jc9qc\" (UID: \"69d994d8-e020-4b34-aa88-586ada797d49\") " pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.236987 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.236966 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 20:26:03.236987 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.236989 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 20:26:03.237176 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.237058 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 20:26:03.237293 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.237278 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 20:26:03.237345 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.237319 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-89gr7\"" Jul 30 20:26:03.250641 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.250618 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl"] Jul 30 20:26:03.321484 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.321410 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:03.326383 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.326362 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.326483 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.326392 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mlh\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-kube-api-access-48mlh\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.326483 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.326412 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8d062-4d8e-4af0-a490-86eeb219cbdd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.326587 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.326485 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.326587 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.326506 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.326587 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.326566 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.326733 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.326604 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.427290 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.427261 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.427456 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.427304 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.427456 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.427352 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.427456 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.427387 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.427456 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.427423 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.427456 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.427452 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mlh\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-kube-api-access-48mlh\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.427710 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.427480 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8d062-4d8e-4af0-a490-86eeb219cbdd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.428203 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.428158 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.430551 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.430519 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.430645 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.430552 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.430645 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.430600 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.431009 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.430990 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8d062-4d8e-4af0-a490-86eeb219cbdd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.436853 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.436828 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.437318 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.437296 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mlh\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-kube-api-access-48mlh\") pod \"istiod-openshift-gateway-7b7995d5cd-rmbgl\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.466405 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.466369 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:03.544472 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.544440 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:03.689337 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.689307 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl"] Jul 30 20:26:03.692475 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:26:03.692448 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda8d062_4d8e_4af0_a490_86eeb219cbdd.slice/crio-46fd4d1636cbfb8d4545417670d5b0a6a7f0e583c29f0fe82d50b2edb602f01c WatchSource:0}: Error finding container 46fd4d1636cbfb8d4545417670d5b0a6a7f0e583c29f0fe82d50b2edb602f01c: Status 404 returned error can't find the container with id 46fd4d1636cbfb8d4545417670d5b0a6a7f0e583c29f0fe82d50b2edb602f01c Jul 30 20:26:03.730195 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.729978 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d994d8-e020-4b34-aa88-586ada797d49-tls-certs\") pod \"model-serving-api-54d89bd5fc-jc9qc\" (UID: \"69d994d8-e020-4b34-aa88-586ada797d49\") " pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.732803 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.732765 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d994d8-e020-4b34-aa88-586ada797d49-tls-certs\") pod \"model-serving-api-54d89bd5fc-jc9qc\" (UID: \"69d994d8-e020-4b34-aa88-586ada797d49\") " pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:03.924275 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:03.924218 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:04.068724 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:04.068675 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jc9qc"] Jul 30 20:26:04.072541 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:26:04.072515 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d994d8_e020_4b34_aa88_586ada797d49.slice/crio-3ef485bb78ca35f494aed4954fe991e2dc37e4072edff0ccc8c424e7ccb25209 WatchSource:0}: Error finding container 3ef485bb78ca35f494aed4954fe991e2dc37e4072edff0ccc8c424e7ccb25209: Status 404 returned error can't find the container with id 3ef485bb78ca35f494aed4954fe991e2dc37e4072edff0ccc8c424e7ccb25209 Jul 30 20:26:04.205468 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:04.205376 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" event={"ID":"eda8d062-4d8e-4af0-a490-86eeb219cbdd","Type":"ContainerStarted","Data":"46fd4d1636cbfb8d4545417670d5b0a6a7f0e583c29f0fe82d50b2edb602f01c"} Jul 30 20:26:04.206702 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:04.206669 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" event={"ID":"69d994d8-e020-4b34-aa88-586ada797d49","Type":"ContainerStarted","Data":"3ef485bb78ca35f494aed4954fe991e2dc37e4072edff0ccc8c424e7ccb25209"} Jul 30 20:26:04.298090 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:04.298054 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:08.013839 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.013731 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:26:08.014137 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.013852 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:26:08.225977 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.225884 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" event={"ID":"c5913a92-4b24-439d-bcbf-810625bf88d0","Type":"ContainerStarted","Data":"4ebbe3f8891b90721b9fc550b3a05144ec368d4ca21df58b530a61db2ce829a2"} Jul 30 20:26:08.226149 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.225983 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:08.227540 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.227512 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" event={"ID":"eda8d062-4d8e-4af0-a490-86eeb219cbdd","Type":"ContainerStarted","Data":"84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008"} Jul 30 20:26:08.227656 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.227582 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:08.228981 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.228954 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" event={"ID":"69d994d8-e020-4b34-aa88-586ada797d49","Type":"ContainerStarted","Data":"5eeed5881bf39f771c8497e530b5c419ed04a96b148873cd1b7ff3b3c407f406"} Jul 30 20:26:08.229091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.228998 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:08.230369 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.230347 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" event={"ID":"f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9","Type":"ContainerStarted","Data":"135a9c3403727e86a2924c9d5b199e634d05ad2928514366c7449a0799f19b74"} Jul 30 20:26:08.266062 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.266009 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" podStartSLOduration=2.263094975 podStartE2EDuration="7.265992992s" totalImagesPullingTime="5.002898017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:08.264563495 +0000 UTC m=+294.524876018" watchObservedRunningTime="2026-07-30 20:26:08.265992992 +0000 UTC m=+294.526305496" Jul 30 20:26:08.292556 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.292498 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55c4ff7d78-ch2mn" podStartSLOduration=2.611857028 podStartE2EDuration="7.292478117s" totalImagesPullingTime="4.680621089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:08.291058939 +0000 UTC m=+294.551371466" watchObservedRunningTime="2026-07-30 20:26:08.292478117 +0000 UTC m=+294.552790646" Jul 30 20:26:08.314920 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.314747 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" podStartSLOduration=0.996810731 podStartE2EDuration="5.314725187s" totalImagesPullingTime="4.317914456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:08.313283258 +0000 UTC m=+294.573595783" watchObservedRunningTime="2026-07-30 20:26:08.314725187 +0000 UTC m=+294.575037694" Jul 30 20:26:08.345109 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.345055 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" podStartSLOduration=2.467876609 podStartE2EDuration="6.345037743s" totalImagesPullingTime="3.877161134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:08.344183731 +0000 UTC m=+294.604496258" watchObservedRunningTime="2026-07-30 20:26:08.345037743 +0000 UTC m=+294.605350269" Jul 30 20:26:08.843535 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.843498 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:08.933353 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:08.933308 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:09.235888 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.235847 2537 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-rmbgl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 20:26:09.236265 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.235919 2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" podUID="eda8d062-4d8e-4af0-a490-86eeb219cbdd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 20:26:09.558731 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.558510 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94"] Jul 30 20:26:09.561908 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.561883 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.564435 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.564405 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-42jxm\"" Jul 30 20:26:09.574158 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.574128 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94"] Jul 30 20:26:09.593756 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.593714 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:09.683134 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683098 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683134 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683139 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af910c78-fd9c-4dfc-96f6-876207dd3f33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683334 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683165 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683334 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683230 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683334 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683264 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683334 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683283 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683334 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683305 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683334 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683322 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.683514 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.683394 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8km\" (UniqueName: \"kubernetes.io/projected/af910c78-fd9c-4dfc-96f6-876207dd3f33-kube-api-access-2v8km\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784698 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784658 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784698 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784698 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784715 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784734 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784813 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8km\" (UniqueName: \"kubernetes.io/projected/af910c78-fd9c-4dfc-96f6-876207dd3f33-kube-api-access-2v8km\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784832 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784854 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af910c78-fd9c-4dfc-96f6-876207dd3f33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784885 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.784953 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.784919 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.785531 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.785504 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.785673 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.785647 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.785858 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.785144 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.785961 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.785843 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.786137 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.786101 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af910c78-fd9c-4dfc-96f6-876207dd3f33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.791692 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.791662 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.791925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.791899 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.794636 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.794608 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af910c78-fd9c-4dfc-96f6-876207dd3f33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.794848 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.794795 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8km\" (UniqueName: \"kubernetes.io/projected/af910c78-fd9c-4dfc-96f6-876207dd3f33-kube-api-access-2v8km\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799czqq94\" (UID: \"af910c78-fd9c-4dfc-96f6-876207dd3f33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.795024 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.794998 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:09.873574 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.873544 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:09.904407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:09.904365 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:10.006331 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:10.006305 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94"] Jul 30 20:26:10.008687 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:26:10.008660 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf910c78_fd9c_4dfc_96f6_876207dd3f33.slice/crio-3f0a1eeddd8dbd035863b20e634446a9f48d0821b3dde7a827fcdaa12476ac83 WatchSource:0}: Error finding container 3f0a1eeddd8dbd035863b20e634446a9f48d0821b3dde7a827fcdaa12476ac83: Status 404 returned error can't find the container with id 3f0a1eeddd8dbd035863b20e634446a9f48d0821b3dde7a827fcdaa12476ac83 Jul 30 20:26:10.239016 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:10.238926 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" event={"ID":"af910c78-fd9c-4dfc-96f6-876207dd3f33","Type":"ContainerStarted","Data":"3f0a1eeddd8dbd035863b20e634446a9f48d0821b3dde7a827fcdaa12476ac83"} Jul 30 20:26:11.830752 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.830711 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:11.965342 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.965271 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:11.966222 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.966199 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5fr66"] Jul 30 20:26:11.973462 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.973430 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd"] Jul 30 20:26:11.973579 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.973566 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:11.976941 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.976911 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:11.979453 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.979432 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 30 20:26:11.980042 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.979718 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gdfbz\"" Jul 30 20:26:11.980407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.980387 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bzdb9\"" Jul 30 20:26:11.980635 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.980574 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 30 20:26:11.994887 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:11.994861 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5fr66"] Jul 30 20:26:12.016606 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.016511 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd"] Jul 30 20:26:12.057679 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.057626 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:12.107656 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.107618 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c4c773-765d-4d42-bc81-1c981a68da0f-cert\") pod \"kserve-controller-manager-d6f4b65b8-5fr66\" (UID: \"e2c4c773-765d-4d42-bc81-1c981a68da0f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:12.107835 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.107664 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5sr\" (UniqueName: \"kubernetes.io/projected/d2d73de9-5c72-4e81-bfe4-4b68375853b7-kube-api-access-lw5sr\") pod \"llmisvc-controller-manager-777c6bd7d5-zkpmd\" (UID: \"d2d73de9-5c72-4e81-bfe4-4b68375853b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:12.107835 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.107698 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d73de9-5c72-4e81-bfe4-4b68375853b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zkpmd\" (UID: \"d2d73de9-5c72-4e81-bfe4-4b68375853b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:12.107835 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.107754 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsf9\" (UniqueName: \"kubernetes.io/projected/e2c4c773-765d-4d42-bc81-1c981a68da0f-kube-api-access-hbsf9\") pod \"kserve-controller-manager-d6f4b65b8-5fr66\" (UID: \"e2c4c773-765d-4d42-bc81-1c981a68da0f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:12.209471 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.209432 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c4c773-765d-4d42-bc81-1c981a68da0f-cert\") pod \"kserve-controller-manager-d6f4b65b8-5fr66\" (UID: \"e2c4c773-765d-4d42-bc81-1c981a68da0f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:12.209654 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.209481 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5sr\" (UniqueName: \"kubernetes.io/projected/d2d73de9-5c72-4e81-bfe4-4b68375853b7-kube-api-access-lw5sr\") pod \"llmisvc-controller-manager-777c6bd7d5-zkpmd\" (UID: \"d2d73de9-5c72-4e81-bfe4-4b68375853b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:12.209654 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.209512 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d73de9-5c72-4e81-bfe4-4b68375853b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zkpmd\" (UID: \"d2d73de9-5c72-4e81-bfe4-4b68375853b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:12.209654 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.209547 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsf9\" (UniqueName: \"kubernetes.io/projected/e2c4c773-765d-4d42-bc81-1c981a68da0f-kube-api-access-hbsf9\") pod \"kserve-controller-manager-d6f4b65b8-5fr66\" (UID: \"e2c4c773-765d-4d42-bc81-1c981a68da0f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:12.209654 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:26:12.209620 2537 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 30 20:26:12.209911 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:26:12.209693 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4c773-765d-4d42-bc81-1c981a68da0f-cert podName:e2c4c773-765d-4d42-bc81-1c981a68da0f nodeName:}" failed. No retries permitted until 2026-07-30 20:26:12.70967109 +0000 UTC m=+298.969983593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e2c4c773-765d-4d42-bc81-1c981a68da0f-cert") pod "kserve-controller-manager-d6f4b65b8-5fr66" (UID: "e2c4c773-765d-4d42-bc81-1c981a68da0f") : secret "kserve-webhook-server-cert" not found Jul 30 20:26:12.212723 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.212692 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d73de9-5c72-4e81-bfe4-4b68375853b7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zkpmd\" (UID: \"d2d73de9-5c72-4e81-bfe4-4b68375853b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:12.229037 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.228997 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsf9\" (UniqueName: \"kubernetes.io/projected/e2c4c773-765d-4d42-bc81-1c981a68da0f-kube-api-access-hbsf9\") pod \"kserve-controller-manager-d6f4b65b8-5fr66\" (UID: \"e2c4c773-765d-4d42-bc81-1c981a68da0f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:12.235481 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.235453 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:12.242553 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.242495 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5sr\" (UniqueName: \"kubernetes.io/projected/d2d73de9-5c72-4e81-bfe4-4b68375853b7-kube-api-access-lw5sr\") pod \"llmisvc-controller-manager-777c6bd7d5-zkpmd\" (UID: \"d2d73de9-5c72-4e81-bfe4-4b68375853b7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:12.243443 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.243399 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:12.297803 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.297752 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:12.349819 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.349753 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:12.510287 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.510180 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:26:12.716069 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.716030 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c4c773-765d-4d42-bc81-1c981a68da0f-cert\") pod \"kserve-controller-manager-d6f4b65b8-5fr66\" (UID: \"e2c4c773-765d-4d42-bc81-1c981a68da0f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:12.718526 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.718499 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2c4c773-765d-4d42-bc81-1c981a68da0f-cert\") pod \"kserve-controller-manager-d6f4b65b8-5fr66\" (UID: \"e2c4c773-765d-4d42-bc81-1c981a68da0f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:12.801055 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.801021 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:26:12.801164 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.801093 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:26:12.801164 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.801122 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:26:12.858717 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.858694 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd"] Jul 30 20:26:12.860826 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:26:12.860797 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2d73de9_5c72_4e81_bfe4_4b68375853b7.slice/crio-44b4cebeb633dd836553b05c5a64da44c0588fff07c9b639e5e3dc315840f13f WatchSource:0}: Error finding container 44b4cebeb633dd836553b05c5a64da44c0588fff07c9b639e5e3dc315840f13f: Status 404 returned error can't find the container with id 44b4cebeb633dd836553b05c5a64da44c0588fff07c9b639e5e3dc315840f13f Jul 30 20:26:12.889382 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:12.889340 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:13.051571 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.051521 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5fr66"] Jul 30 20:26:13.054168 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:26:13.054140 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c4c773_765d_4d42_bc81_1c981a68da0f.slice/crio-cc9ce18e53bbcb2eeb9c4f2e3e71abd60621a4c5fb14c0429aa3877fe7e03415 WatchSource:0}: Error finding container cc9ce18e53bbcb2eeb9c4f2e3e71abd60621a4c5fb14c0429aa3877fe7e03415: Status 404 returned error can't find the container with id cc9ce18e53bbcb2eeb9c4f2e3e71abd60621a4c5fb14c0429aa3877fe7e03415 Jul 30 20:26:13.252860 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.252810 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" event={"ID":"e2c4c773-765d-4d42-bc81-1c981a68da0f","Type":"ContainerStarted","Data":"cc9ce18e53bbcb2eeb9c4f2e3e71abd60621a4c5fb14c0429aa3877fe7e03415"} Jul 30 20:26:13.253914 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.253886 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" event={"ID":"d2d73de9-5c72-4e81-bfe4-4b68375853b7","Type":"ContainerStarted","Data":"44b4cebeb633dd836553b05c5a64da44c0588fff07c9b639e5e3dc315840f13f"} Jul 30 20:26:13.255336 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.255316 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" event={"ID":"af910c78-fd9c-4dfc-96f6-876207dd3f33","Type":"ContainerStarted","Data":"36d3abb8ee0cd3e1fc2aabc4f1a5c16c4da9d9c199aa6cd4dc11ac5b050eaf88"} Jul 30 20:26:13.286189 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.286137 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" podStartSLOduration=1.496163151 podStartE2EDuration="4.286118945s" totalImagesPullingTime="2.789955794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:13.284652967 +0000 UTC m=+299.544965493" watchObservedRunningTime="2026-07-30 20:26:13.286118945 +0000 UTC m=+299.546431470" Jul 30 20:26:13.600596 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.600560 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl"] Jul 30 20:26:13.600857 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.600830 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" podUID="eda8d062-4d8e-4af0-a490-86eeb219cbdd" containerName="discovery" containerID="cri-o://84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008" gracePeriod=30 Jul 30 20:26:13.873690 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.873641 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:13.881718 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:13.881687 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:14.017444 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.017416 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:14.129309 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.129125 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-ca-configmap\") pod \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " Jul 30 20:26:14.129309 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.129179 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8d062-4d8e-4af0-a490-86eeb219cbdd-local-certs\") pod \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " Jul 30 20:26:14.129309 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.129208 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-dns-cert\") pod \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " Jul 30 20:26:14.129309 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.129251 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48mlh\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-kube-api-access-48mlh\") pod \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " Jul 30 20:26:14.129309 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.129292 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-cacerts\") pod \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " Jul 30 20:26:14.132258 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.129317 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-kubeconfig\") pod \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " Jul 30 20:26:14.132258 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.129370 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-token\") pod \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\" (UID: \"eda8d062-4d8e-4af0-a490-86eeb219cbdd\") " Jul 30 20:26:14.132258 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.130981 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "eda8d062-4d8e-4af0-a490-86eeb219cbdd" (UID: "eda8d062-4d8e-4af0-a490-86eeb219cbdd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 20:26:14.132258 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.132157 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-token" (OuterVolumeSpecName: "istio-token") pod "eda8d062-4d8e-4af0-a490-86eeb219cbdd" (UID: "eda8d062-4d8e-4af0-a490-86eeb219cbdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:26:14.140218 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.139792 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda8d062-4d8e-4af0-a490-86eeb219cbdd-local-certs" (OuterVolumeSpecName: "local-certs") pod "eda8d062-4d8e-4af0-a490-86eeb219cbdd" (UID: "eda8d062-4d8e-4af0-a490-86eeb219cbdd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:26:14.140492 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.140465 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-kube-api-access-48mlh" (OuterVolumeSpecName: "kube-api-access-48mlh") pod "eda8d062-4d8e-4af0-a490-86eeb219cbdd" (UID: "eda8d062-4d8e-4af0-a490-86eeb219cbdd"). InnerVolumeSpecName "kube-api-access-48mlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:26:14.141163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.141138 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "eda8d062-4d8e-4af0-a490-86eeb219cbdd" (UID: "eda8d062-4d8e-4af0-a490-86eeb219cbdd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:26:14.142448 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.142404 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "eda8d062-4d8e-4af0-a490-86eeb219cbdd" (UID: "eda8d062-4d8e-4af0-a490-86eeb219cbdd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:26:14.143044 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.143008 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-cacerts" (OuterVolumeSpecName: "cacerts") pod "eda8d062-4d8e-4af0-a490-86eeb219cbdd" (UID: "eda8d062-4d8e-4af0-a490-86eeb219cbdd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 20:26:14.230636 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.230605 2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-token\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:26:14.230636 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.230634 2537 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-ca-configmap\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:26:14.230636 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.230645 2537 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eda8d062-4d8e-4af0-a490-86eeb219cbdd-local-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:26:14.230870 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.230653 2537 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-csr-dns-cert\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:26:14.230870 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.230662 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48mlh\" (UniqueName: \"kubernetes.io/projected/eda8d062-4d8e-4af0-a490-86eeb219cbdd-kube-api-access-48mlh\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:26:14.230870 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.230670 2537 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-cacerts\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:26:14.230870 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.230679 2537 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eda8d062-4d8e-4af0-a490-86eeb219cbdd-istio-kubeconfig\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:26:14.263332 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.263292 2537 generic.go:356] "Generic (PLEG): container finished" podID="eda8d062-4d8e-4af0-a490-86eeb219cbdd" containerID="84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008" exitCode=0 Jul 30 20:26:14.263497 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.263375 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" Jul 30 20:26:14.263497 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.263400 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" event={"ID":"eda8d062-4d8e-4af0-a490-86eeb219cbdd","Type":"ContainerDied","Data":"84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008"} Jul 30 20:26:14.263497 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.263454 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl" event={"ID":"eda8d062-4d8e-4af0-a490-86eeb219cbdd","Type":"ContainerDied","Data":"46fd4d1636cbfb8d4545417670d5b0a6a7f0e583c29f0fe82d50b2edb602f01c"} Jul 30 20:26:14.263497 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.263480 2537 scope.go:117] "RemoveContainer" containerID="84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008" Jul 30 20:26:14.264337 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.264290 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:14.265453 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.265431 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799czqq94" Jul 30 20:26:14.271347 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.271329 2537 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 20:26:14.276715 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.276694 2537 scope.go:117] "RemoveContainer" containerID="84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008" Jul 30 20:26:14.277031 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:26:14.277003 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008\": container with ID starting with 84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008 not found: ID does not exist" containerID="84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008" Jul 30 20:26:14.277128 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.277040 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008"} err="failed to get container status \"84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008\": rpc error: code = NotFound desc = could not find container \"84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008\": container with ID starting with 84a283aad3c12c0baa054973548116b7e6bf791bd4b68d7135a452b8a23a6008 not found: ID does not exist" Jul 30 20:26:14.355691 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.355663 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl"] Jul 30 20:26:14.366042 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:14.366016 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rmbgl"] Jul 30 20:26:16.129138 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:16.129110 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eda8d062-4d8e-4af0-a490-86eeb219cbdd" path="/var/lib/kubelet/pods/eda8d062-4d8e-4af0-a490-86eeb219cbdd/volumes" Jul 30 20:26:16.275688 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:16.275651 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" event={"ID":"d2d73de9-5c72-4e81-bfe4-4b68375853b7","Type":"ContainerStarted","Data":"e23e5f7eb49dedb0b7691cd4b446ad3ea2ac35294cc8d0fce1217acc7419a241"} Jul 30 20:26:16.275886 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:16.275717 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:16.277163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:16.277129 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" event={"ID":"e2c4c773-765d-4d42-bc81-1c981a68da0f","Type":"ContainerStarted","Data":"9dc07b4ebc338177a308d757692d7fe8c030b46cafe6ede1578e2735a3438705"} Jul 30 20:26:16.277305 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:16.277245 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:26:16.292937 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:16.292894 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" podStartSLOduration=2.055294596 podStartE2EDuration="5.292880006s" totalImagesPullingTime="3.23758541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:16.291994786 +0000 UTC m=+302.552307313" watchObservedRunningTime="2026-07-30 20:26:16.292880006 +0000 UTC m=+302.553192546" Jul 30 20:26:16.310069 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:16.310015 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" podStartSLOduration=2.26992995 podStartE2EDuration="5.309998625s" totalImagesPullingTime="3.040068675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:26:16.30966487 +0000 UTC m=+302.569977399" watchObservedRunningTime="2026-07-30 20:26:16.309998625 +0000 UTC m=+302.570311151" Jul 30 20:26:19.236878 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:19.236847 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2m8b5" Jul 30 20:26:19.238685 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:19.238666 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jc9qc" Jul 30 20:26:47.282483 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:47.282454 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zkpmd" Jul 30 20:26:47.285484 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:26:47.285465 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5fr66" Jul 30 20:27:02.863910 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:02.863873 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:02.949380 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:02.949345 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:03.797547 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:03.797507 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:03.872397 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:03.872364 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:22.198244 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:22.198203 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:23.001017 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:23.000977 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:23.789753 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:23.789717 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:26.050493 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.050439 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:26.838010 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.837976 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r"] Jul 30 20:27:26.838306 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.838294 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda8d062-4d8e-4af0-a490-86eeb219cbdd" containerName="discovery" Jul 30 20:27:26.838349 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.838308 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda8d062-4d8e-4af0-a490-86eeb219cbdd" containerName="discovery" Jul 30 20:27:26.838383 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.838355 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda8d062-4d8e-4af0-a490-86eeb219cbdd" containerName="discovery" Jul 30 20:27:26.842574 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.842553 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:26.846112 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.846083 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 20:27:26.846249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.846083 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56ccm\"" Jul 30 20:27:26.846249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.846178 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 20:27:26.863331 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:26.863298 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r"] Jul 30 20:27:27.020034 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.019991 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f048676-a363-4203-8979-7857de7d08b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:27.020213 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.020047 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvr5\" (UniqueName: \"kubernetes.io/projected/2f048676-a363-4203-8979-7857de7d08b1-kube-api-access-vmvr5\") pod \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:27.121407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.121375 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvr5\" (UniqueName: \"kubernetes.io/projected/2f048676-a363-4203-8979-7857de7d08b1-kube-api-access-vmvr5\") pod \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:27.121809 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.121470 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f048676-a363-4203-8979-7857de7d08b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:27.121914 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.121895 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f048676-a363-4203-8979-7857de7d08b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:27.129698 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.129678 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvr5\" (UniqueName: \"kubernetes.io/projected/2f048676-a363-4203-8979-7857de7d08b1-kube-api-access-vmvr5\") pod \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:27.152735 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.152692 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:27.287639 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.287606 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r"] Jul 30 20:27:27.292367 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:27:27.292339 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f048676_a363_4203_8979_7857de7d08b1.slice/crio-ede6943e6f07396922dd4a125c1c2f048e5811167c55d9b6694099685ca5dc50 WatchSource:0}: Error finding container ede6943e6f07396922dd4a125c1c2f048e5811167c55d9b6694099685ca5dc50: Status 404 returned error can't find the container with id ede6943e6f07396922dd4a125c1c2f048e5811167c55d9b6694099685ca5dc50 Jul 30 20:27:27.294933 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.294913 2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 20:27:27.406337 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.406250 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:27.556339 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:27.556307 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" event={"ID":"2f048676-a363-4203-8979-7857de7d08b1","Type":"ContainerStarted","Data":"ede6943e6f07396922dd4a125c1c2f048e5811167c55d9b6694099685ca5dc50"} Jul 30 20:27:28.751755 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:28.751721 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2"] Jul 30 20:27:28.755546 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:28.755522 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:28.758354 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:28.758330 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kj2xm\"" Jul 30 20:27:28.765959 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:28.765884 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2"] Jul 30 20:27:28.938296 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:28.938245 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bd4\" (UniqueName: \"kubernetes.io/projected/b1734288-6641-4a60-bacc-767b23c44889-kube-api-access-c6bd4\") pod \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" (UID: \"b1734288-6641-4a60-bacc-767b23c44889\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:29.006731 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:29.006650 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:29.039746 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:29.039668 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bd4\" (UniqueName: \"kubernetes.io/projected/b1734288-6641-4a60-bacc-767b23c44889-kube-api-access-c6bd4\") pod \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" (UID: \"b1734288-6641-4a60-bacc-767b23c44889\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:29.052089 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:29.052059 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bd4\" (UniqueName: \"kubernetes.io/projected/b1734288-6641-4a60-bacc-767b23c44889-kube-api-access-c6bd4\") pod \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" (UID: \"b1734288-6641-4a60-bacc-767b23c44889\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:29.073766 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:29.073713 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:29.255516 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:29.255460 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:29.413818 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:29.413764 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:29.881620 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:29.881457 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2"] Jul 30 20:27:29.884863 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:27:29.884834 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1734288_6641_4a60_bacc_767b23c44889.slice/crio-9d1ed4a154555b89f87b0aa0f0a001326e1072308642c7150f66960ed6261400 WatchSource:0}: Error finding container 9d1ed4a154555b89f87b0aa0f0a001326e1072308642c7150f66960ed6261400: Status 404 returned error can't find the container with id 9d1ed4a154555b89f87b0aa0f0a001326e1072308642c7150f66960ed6261400 Jul 30 20:27:30.574160 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:30.574125 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" event={"ID":"b1734288-6641-4a60-bacc-767b23c44889","Type":"ContainerStarted","Data":"9d1ed4a154555b89f87b0aa0f0a001326e1072308642c7150f66960ed6261400"} Jul 30 20:27:31.456248 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:31.456221 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:31.551303 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:31.551267 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:31.580934 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:31.580900 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" event={"ID":"2f048676-a363-4203-8979-7857de7d08b1","Type":"ContainerStarted","Data":"aea2a35c1cc7f3e2778b7412302880cbe801b82f33ee9710fad53eaffab8c130"} Jul 30 20:27:31.581117 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:31.581005 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:31.608102 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:31.608030 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" podStartSLOduration=1.470495033 podStartE2EDuration="5.608008866s" totalImagesPullingTime="4.137513833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:31.60509665 +0000 UTC m=+377.865409177" watchObservedRunningTime="2026-07-30 20:27:31.608008866 +0000 UTC m=+377.868321394" Jul 30 20:27:32.069036 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:32.068996 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:32.182644 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:32.182607 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:32.271404 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:32.271370 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:32.589091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:32.589050 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" event={"ID":"b1734288-6641-4a60-bacc-767b23c44889","Type":"ContainerStarted","Data":"403ab4ea5e45a1669ab1e28d83c35e39eacfc0f96ec9bb78058db8a3196ca985"} Jul 30 20:27:32.610222 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:32.610167 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" podStartSLOduration=2.245389372 podStartE2EDuration="4.610151007s" totalImagesPullingTime="2.364761635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:32.609929918 +0000 UTC m=+378.870242445" watchObservedRunningTime="2026-07-30 20:27:32.610151007 +0000 UTC m=+378.870463532" Jul 30 20:27:32.805163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:32.805104 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:32.904541 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:32.904478 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:33.593654 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.593624 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:33.872753 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.872721 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-w776m"] Jul 30 20:27:33.876055 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.876040 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" Jul 30 20:27:33.879421 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.879399 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 30 20:27:33.880231 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.880213 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-k9xnk\"" Jul 30 20:27:33.882984 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.882958 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrlr\" (UniqueName: \"kubernetes.io/projected/52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e-kube-api-access-cbrlr\") pod \"dns-operator-controller-manager-5985564776-w776m\" (UID: \"52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" Jul 30 20:27:33.912670 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.912639 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-w776m"] Jul 30 20:27:33.983860 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.983816 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrlr\" (UniqueName: \"kubernetes.io/projected/52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e-kube-api-access-cbrlr\") pod \"dns-operator-controller-manager-5985564776-w776m\" (UID: \"52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" Jul 30 20:27:33.999592 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:33.999562 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrlr\" (UniqueName: \"kubernetes.io/projected/52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e-kube-api-access-cbrlr\") pod \"dns-operator-controller-manager-5985564776-w776m\" (UID: \"52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" Jul 30 20:27:34.185631 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.185550 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" Jul 30 20:27:34.317282 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.317254 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-w776m"] Jul 30 20:27:34.319505 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:27:34.319476 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52733b6b_3ff3_4c0d_8fbe_2a7f812d2a3e.slice/crio-1e35d63d04649195b3ee290138a165738cb28efcb9fded286e5353572ad1109c WatchSource:0}: Error finding container 1e35d63d04649195b3ee290138a165738cb28efcb9fded286e5353572ad1109c: Status 404 returned error can't find the container with id 1e35d63d04649195b3ee290138a165738cb28efcb9fded286e5353572ad1109c Jul 30 20:27:34.355071 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.355039 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58"] Jul 30 20:27:34.355321 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.355282 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:34.359486 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.359469 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.362218 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.362193 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7ctr8\"" Jul 30 20:27:34.362533 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.362511 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 30 20:27:34.362609 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.362518 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 30 20:27:34.368257 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.368229 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58"] Jul 30 20:27:34.387281 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.387251 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15ce5183-3608-4f26-9322-3e6677a6d21f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.387450 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.387332 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/15ce5183-3608-4f26-9322-3e6677a6d21f-kube-api-access-bsk5z\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.387450 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.387384 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/15ce5183-3608-4f26-9322-3e6677a6d21f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.488159 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.488065 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15ce5183-3608-4f26-9322-3e6677a6d21f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.488159 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.488128 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/15ce5183-3608-4f26-9322-3e6677a6d21f-kube-api-access-bsk5z\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.488351 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.488173 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/15ce5183-3608-4f26-9322-3e6677a6d21f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.488351 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:34.488318 2537 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 30 20:27:34.488425 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:34.488416 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ce5183-3608-4f26-9322-3e6677a6d21f-plugin-serving-cert podName:15ce5183-3608-4f26-9322-3e6677a6d21f nodeName:}" failed. No retries permitted until 2026-07-30 20:27:34.988380052 +0000 UTC m=+381.248692556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/15ce5183-3608-4f26-9322-3e6677a6d21f-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-ppb58" (UID: "15ce5183-3608-4f26-9322-3e6677a6d21f") : secret "plugin-serving-cert" not found Jul 30 20:27:34.488947 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.488924 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15ce5183-3608-4f26-9322-3e6677a6d21f-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.501468 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.501434 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsk5z\" (UniqueName: \"kubernetes.io/projected/15ce5183-3608-4f26-9322-3e6677a6d21f-kube-api-access-bsk5z\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.599252 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.599209 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" event={"ID":"52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e","Type":"ContainerStarted","Data":"1e35d63d04649195b3ee290138a165738cb28efcb9fded286e5353572ad1109c"} Jul 30 20:27:34.743116 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.743003 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:34.991407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.991365 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/15ce5183-3608-4f26-9322-3e6677a6d21f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:34.993881 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:34.993831 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/15ce5183-3608-4f26-9322-3e6677a6d21f-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ppb58\" (UID: \"15ce5183-3608-4f26-9322-3e6677a6d21f\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:35.271634 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:35.271552 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" Jul 30 20:27:35.404203 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:35.404104 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58"] Jul 30 20:27:35.406966 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:27:35.406922 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15ce5183_3608_4f26_9322_3e6677a6d21f.slice/crio-447f8053c71de6b3ed4e68c980b3146e8d4c8130f1fe1b6311674e01f1a261a1 WatchSource:0}: Error finding container 447f8053c71de6b3ed4e68c980b3146e8d4c8130f1fe1b6311674e01f1a261a1: Status 404 returned error can't find the container with id 447f8053c71de6b3ed4e68c980b3146e8d4c8130f1fe1b6311674e01f1a261a1 Jul 30 20:27:35.605199 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:35.605107 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" event={"ID":"15ce5183-3608-4f26-9322-3e6677a6d21f","Type":"ContainerStarted","Data":"447f8053c71de6b3ed4e68c980b3146e8d4c8130f1fe1b6311674e01f1a261a1"} Jul 30 20:27:36.611757 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:36.611724 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" event={"ID":"52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e","Type":"ContainerStarted","Data":"01d3b401b1199da455469466ce8d462d18454573f2121d96ac8f0e3f97d9383a"} Jul 30 20:27:36.612177 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:36.611831 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" Jul 30 20:27:36.630996 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:36.630945 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" podStartSLOduration=1.7486502879999999 podStartE2EDuration="3.630925951s" totalImagesPullingTime="1.882275663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:36.628712058 +0000 UTC m=+382.889024580" watchObservedRunningTime="2026-07-30 20:27:36.630925951 +0000 UTC m=+382.891238477" Jul 30 20:27:37.213622 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:37.213573 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:37.280876 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:37.280841 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:37.602514 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:37.602429 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:37.667211 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:37.667171 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:38.006408 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:38.006374 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:38.070922 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:38.070885 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:40.632474 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:40.632438 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" event={"ID":"15ce5183-3608-4f26-9322-3e6677a6d21f","Type":"ContainerStarted","Data":"40689f02a10f2f68fe32a3fb3418b73d642c6e26148a974ecd7197073dc7b2b4"} Jul 30 20:27:40.651814 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:40.651740 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ppb58" podStartSLOduration=1.581756358 podStartE2EDuration="6.651722127s" totalImagesPullingTime="5.069965769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:40.650706477 +0000 UTC m=+386.911019002" watchObservedRunningTime="2026-07-30 20:27:40.651722127 +0000 UTC m=+386.912034692" Jul 30 20:27:42.592124 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:42.592087 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:43.336684 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:43.336652 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:43.458922 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:43.458885 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:43.562225 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:43.562183 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:44.213671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.213638 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7"] Jul 30 20:27:44.217190 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.217174 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.232931 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.232907 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7"] Jul 30 20:27:44.250457 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.250426 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:44.269765 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.269731 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b62a14-f1f7-4533-b64e-3fda0956bd4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.269930 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.269793 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkhw\" (UniqueName: \"kubernetes.io/projected/89b62a14-f1f7-4533-b64e-3fda0956bd4b-kube-api-access-gfkhw\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.355076 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.355031 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:44.371106 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.371073 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b62a14-f1f7-4533-b64e-3fda0956bd4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.371257 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.371124 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkhw\" (UniqueName: \"kubernetes.io/projected/89b62a14-f1f7-4533-b64e-3fda0956bd4b-kube-api-access-gfkhw\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.371457 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.371436 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b62a14-f1f7-4533-b64e-3fda0956bd4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.381293 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.381266 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkhw\" (UniqueName: \"kubernetes.io/projected/89b62a14-f1f7-4533-b64e-3fda0956bd4b-kube-api-access-gfkhw\") pod \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.418577 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.418541 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7"] Jul 30 20:27:44.418829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.418817 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:44.436224 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.436198 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r"] Jul 30 20:27:44.436423 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.436399 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" podUID="2f048676-a363-4203-8979-7857de7d08b1" containerName="manager" containerID="cri-o://aea2a35c1cc7f3e2778b7412302880cbe801b82f33ee9710fad53eaffab8c130" gracePeriod=2 Jul 30 20:27:44.448441 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.441662 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:44.448441 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.442511 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r"] Jul 30 20:27:44.463152 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.463109 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc"] Jul 30 20:27:44.463625 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.463602 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f048676-a363-4203-8979-7857de7d08b1" containerName="manager" Jul 30 20:27:44.463766 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.463628 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f048676-a363-4203-8979-7857de7d08b1" containerName="manager" Jul 30 20:27:44.463766 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.463727 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f048676-a363-4203-8979-7857de7d08b1" containerName="manager" Jul 30 20:27:44.468706 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.468153 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.470866 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.469857 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7"] Jul 30 20:27:44.482795 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.480111 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7"] Jul 30 20:27:44.482795 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.482398 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc"] Jul 30 20:27:44.503480 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.503427 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2"] Jul 30 20:27:44.503707 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.503673 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" podUID="b1734288-6641-4a60-bacc-767b23c44889" containerName="manager" containerID="cri-o://403ab4ea5e45a1669ab1e28d83c35e39eacfc0f96ec9bb78058db8a3196ca985" gracePeriod=2 Jul 30 20:27:44.504836 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.504817 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2"] Jul 30 20:27:44.505823 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.505224 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:44.507998 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.507965 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:44.511493 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.511468 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="b1734288-6641-4a60-bacc-767b23c44889" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:44.512656 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.512636 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj"] Jul 30 20:27:44.513125 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.513109 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1734288-6641-4a60-bacc-767b23c44889" containerName="manager" Jul 30 20:27:44.513213 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.513126 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1734288-6641-4a60-bacc-767b23c44889" containerName="manager" Jul 30 20:27:44.513213 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.513183 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1734288-6641-4a60-bacc-767b23c44889" containerName="manager" Jul 30 20:27:44.521054 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.521038 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" Jul 30 20:27:44.523513 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.523486 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:44.529561 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.529532 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj"] Jul 30 20:27:44.542743 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.542697 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="b1734288-6641-4a60-bacc-767b23c44889" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:44.577397 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.577372 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlgxp\" (UniqueName: \"kubernetes.io/projected/abb37ac3-96f5-4880-b909-2605d9452ff9-kube-api-access-wlgxp\") pod \"limitador-operator-controller-manager-6c549b6d85-f8ztj\" (UID: \"abb37ac3-96f5-4880-b909-2605d9452ff9\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" Jul 30 20:27:44.577514 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.577404 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a701e93-3bae-4371-9349-b6747dacb652-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-c5slc\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.577514 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.577480 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq77\" (UniqueName: \"kubernetes.io/projected/4a701e93-3bae-4371-9349-b6747dacb652-kube-api-access-7nq77\") pod \"kuadrant-operator-controller-manager-68cf589589-c5slc\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.652029 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.651989 2537 generic.go:356] "Generic (PLEG): container finished" podID="2f048676-a363-4203-8979-7857de7d08b1" containerID="aea2a35c1cc7f3e2778b7412302880cbe801b82f33ee9710fad53eaffab8c130" exitCode=0 Jul 30 20:27:44.653997 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.653967 2537 generic.go:356] "Generic (PLEG): container finished" podID="b1734288-6641-4a60-bacc-767b23c44889" containerID="403ab4ea5e45a1669ab1e28d83c35e39eacfc0f96ec9bb78058db8a3196ca985" exitCode=0 Jul 30 20:27:44.678318 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.678272 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq77\" (UniqueName: \"kubernetes.io/projected/4a701e93-3bae-4371-9349-b6747dacb652-kube-api-access-7nq77\") pod \"kuadrant-operator-controller-manager-68cf589589-c5slc\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.678694 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.678324 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgxp\" (UniqueName: \"kubernetes.io/projected/abb37ac3-96f5-4880-b909-2605d9452ff9-kube-api-access-wlgxp\") pod \"limitador-operator-controller-manager-6c549b6d85-f8ztj\" (UID: \"abb37ac3-96f5-4880-b909-2605d9452ff9\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" Jul 30 20:27:44.678694 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.678364 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a701e93-3bae-4371-9349-b6747dacb652-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-c5slc\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.678912 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.678805 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a701e93-3bae-4371-9349-b6747dacb652-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-c5slc\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.689351 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.689327 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq77\" (UniqueName: \"kubernetes.io/projected/4a701e93-3bae-4371-9349-b6747dacb652-kube-api-access-7nq77\") pod \"kuadrant-operator-controller-manager-68cf589589-c5slc\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.689452 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.689384 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlgxp\" (UniqueName: \"kubernetes.io/projected/abb37ac3-96f5-4880-b909-2605d9452ff9-kube-api-access-wlgxp\") pod \"limitador-operator-controller-manager-6c549b6d85-f8ztj\" (UID: \"abb37ac3-96f5-4880-b909-2605d9452ff9\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" Jul 30 20:27:44.723895 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.723864 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:44.726206 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.726173 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="b1734288-6641-4a60-bacc-767b23c44889" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:44.728102 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.728070 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:44.730895 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.730879 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:44.732689 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.732665 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:44.734654 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:44.734631 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="b1734288-6641-4a60-bacc-767b23c44889" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:44.779131 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.779094 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmvr5\" (UniqueName: \"kubernetes.io/projected/2f048676-a363-4203-8979-7857de7d08b1-kube-api-access-vmvr5\") pod \"2f048676-a363-4203-8979-7857de7d08b1\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " Jul 30 20:27:44.779308 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.779152 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f048676-a363-4203-8979-7857de7d08b1-extensions-socket-volume\") pod \"2f048676-a363-4203-8979-7857de7d08b1\" (UID: \"2f048676-a363-4203-8979-7857de7d08b1\") " Jul 30 20:27:44.779308 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.779180 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6bd4\" (UniqueName: \"kubernetes.io/projected/b1734288-6641-4a60-bacc-767b23c44889-kube-api-access-c6bd4\") pod \"b1734288-6641-4a60-bacc-767b23c44889\" (UID: \"b1734288-6641-4a60-bacc-767b23c44889\") " Jul 30 20:27:44.779648 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.779619 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f048676-a363-4203-8979-7857de7d08b1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2f048676-a363-4203-8979-7857de7d08b1" (UID: "2f048676-a363-4203-8979-7857de7d08b1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:44.781302 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.781277 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f048676-a363-4203-8979-7857de7d08b1-kube-api-access-vmvr5" (OuterVolumeSpecName: "kube-api-access-vmvr5") pod "2f048676-a363-4203-8979-7857de7d08b1" (UID: "2f048676-a363-4203-8979-7857de7d08b1"). InnerVolumeSpecName "kube-api-access-vmvr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:44.781302 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.781290 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1734288-6641-4a60-bacc-767b23c44889-kube-api-access-c6bd4" (OuterVolumeSpecName: "kube-api-access-c6bd4") pod "b1734288-6641-4a60-bacc-767b23c44889" (UID: "b1734288-6641-4a60-bacc-767b23c44889"). InnerVolumeSpecName "kube-api-access-c6bd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:44.871642 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.871605 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:44.876698 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.876674 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" Jul 30 20:27:44.879727 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.879705 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmvr5\" (UniqueName: \"kubernetes.io/projected/2f048676-a363-4203-8979-7857de7d08b1-kube-api-access-vmvr5\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:27:44.879797 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.879734 2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f048676-a363-4203-8979-7857de7d08b1-extensions-socket-volume\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:27:44.879797 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:44.879749 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6bd4\" (UniqueName: \"kubernetes.io/projected/b1734288-6641-4a60-bacc-767b23c44889-kube-api-access-c6bd4\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:27:45.026553 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.026519 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc"] Jul 30 20:27:45.027761 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:27:45.027729 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a701e93_3bae_4371_9349_b6747dacb652.slice/crio-690989bd436102d5bce8078285282f69f1ad1fc8458802e797ce2ac5bbc03b86 WatchSource:0}: Error finding container 690989bd436102d5bce8078285282f69f1ad1fc8458802e797ce2ac5bbc03b86: Status 404 returned error can't find the container with id 690989bd436102d5bce8078285282f69f1ad1fc8458802e797ce2ac5bbc03b86 Jul 30 20:27:45.036837 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.036736 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj"] Jul 30 20:27:45.039757 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:27:45.039728 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb37ac3_96f5_4880_b909_2605d9452ff9.slice/crio-1de89bdfa1f37b1657cfe2c6a1d64b4a3762e9a018ea243afac8a8aab1ec1c14 WatchSource:0}: Error finding container 1de89bdfa1f37b1657cfe2c6a1d64b4a3762e9a018ea243afac8a8aab1ec1c14: Status 404 returned error can't find the container with id 1de89bdfa1f37b1657cfe2c6a1d64b4a3762e9a018ea243afac8a8aab1ec1c14 Jul 30 20:27:45.450828 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:45.450790 2537 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 30 20:27:45.450828 ip-10-0-131-223 kubenswrapper[2537]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-rvxn7_kuadrant-system_89b62a14-f1f7-4533-b64e-3fda0956bd4b_0(3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-rvxn7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4" Netns:"/var/run/netns/15b7b5b3-57a5-4f13-ab6c-92b854b1919f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-rvxn7;K8S_POD_INFRA_CONTAINER_ID=3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4;K8S_POD_UID=89b62a14-f1f7-4533-b64e-3fda0956bd4b" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7/89b62a14-f1f7-4533-b64e-3fda0956bd4b:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7 3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4 network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-6bd956ccf-rvxn7" not found Jul 30 20:27:45.450828 ip-10-0-131-223 kubenswrapper[2537]: ' Jul 30 20:27:45.450828 ip-10-0-131-223 kubenswrapper[2537]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 30 20:27:45.450828 ip-10-0-131-223 kubenswrapper[2537]: > Jul 30 20:27:45.451211 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:45.450858 2537 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 30 20:27:45.451211 ip-10-0-131-223 kubenswrapper[2537]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6bd956ccf-rvxn7_kuadrant-system_89b62a14-f1f7-4533-b64e-3fda0956bd4b_0(3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6bd956ccf-rvxn7 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4" Netns:"/var/run/netns/15b7b5b3-57a5-4f13-ab6c-92b854b1919f" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-6bd956ccf-rvxn7;K8S_POD_INFRA_CONTAINER_ID=3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4;K8S_POD_UID=89b62a14-f1f7-4533-b64e-3fda0956bd4b" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7] networking: [kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7/89b62a14-f1f7-4533-b64e-3fda0956bd4b:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7 3d04c8fb45b54b661e7fbc14de5ab331918bc047558fa8a83dbaef6768576cb4 network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-6bd956ccf-rvxn7" not found Jul 30 20:27:45.451211 ip-10-0-131-223 kubenswrapper[2537]: ' Jul 30 20:27:45.451211 ip-10-0-131-223 kubenswrapper[2537]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 30 20:27:45.451211 ip-10-0-131-223 kubenswrapper[2537]: > pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:45.628621 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.628525 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:45.659579 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.659546 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" event={"ID":"abb37ac3-96f5-4880-b909-2605d9452ff9","Type":"ContainerStarted","Data":"cf222a0cea4eee4b8e9be041f6940f37e989f1f1676401203e4fe68a116fb88c"} Jul 30 20:27:45.659747 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.659597 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" event={"ID":"abb37ac3-96f5-4880-b909-2605d9452ff9","Type":"ContainerStarted","Data":"1de89bdfa1f37b1657cfe2c6a1d64b4a3762e9a018ea243afac8a8aab1ec1c14"} Jul 30 20:27:45.659747 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.659645 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" Jul 30 20:27:45.661245 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.661216 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" event={"ID":"4a701e93-3bae-4371-9349-b6747dacb652","Type":"ContainerStarted","Data":"0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40"} Jul 30 20:27:45.661365 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.661262 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" event={"ID":"4a701e93-3bae-4371-9349-b6747dacb652","Type":"ContainerStarted","Data":"690989bd436102d5bce8078285282f69f1ad1fc8458802e797ce2ac5bbc03b86"} Jul 30 20:27:45.661365 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.661287 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:45.662007 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:45.661987 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="b1734288-6641-4a60-bacc-767b23c44889" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:45.662600 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.662585 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:45.662671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.662612 2537 scope.go:117] "RemoveContainer" containerID="aea2a35c1cc7f3e2778b7412302880cbe801b82f33ee9710fad53eaffab8c130" Jul 30 20:27:45.664006 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:45.663966 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:45.664120 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.664107 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:45.664174 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.664126 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:45.672480 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.672463 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:45.673015 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.673000 2537 scope.go:117] "RemoveContainer" containerID="403ab4ea5e45a1669ab1e28d83c35e39eacfc0f96ec9bb78058db8a3196ca985" Jul 30 20:27:45.684710 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.684669 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" podStartSLOduration=1.684654846 podStartE2EDuration="1.684654846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:45.684261062 +0000 UTC m=+391.944573578" watchObservedRunningTime="2026-07-30 20:27:45.684654846 +0000 UTC m=+391.944967372" Jul 30 20:27:45.686350 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:45.686328 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-hqgn2\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="b1734288-6641-4a60-bacc-767b23c44889" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hqgn2" Jul 30 20:27:45.688016 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.687991 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:45.706363 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.706320 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" podStartSLOduration=1.706304396 podStartE2EDuration="1.706304396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:27:45.704805329 +0000 UTC m=+391.965117856" watchObservedRunningTime="2026-07-30 20:27:45.706304396 +0000 UTC m=+391.966616921" Jul 30 20:27:45.707101 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:45.707078 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-pck2r\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="2f048676-a363-4203-8979-7857de7d08b1" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-pck2r" Jul 30 20:27:45.708996 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:45.708975 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="89b62a14-f1f7-4533-b64e-3fda0956bd4b" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:45.786115 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.786082 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b62a14-f1f7-4533-b64e-3fda0956bd4b-extensions-socket-volume\") pod \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " Jul 30 20:27:45.786294 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.786169 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfkhw\" (UniqueName: \"kubernetes.io/projected/89b62a14-f1f7-4533-b64e-3fda0956bd4b-kube-api-access-gfkhw\") pod \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\" (UID: \"89b62a14-f1f7-4533-b64e-3fda0956bd4b\") " Jul 30 20:27:45.786408 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.786386 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b62a14-f1f7-4533-b64e-3fda0956bd4b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "89b62a14-f1f7-4533-b64e-3fda0956bd4b" (UID: "89b62a14-f1f7-4533-b64e-3fda0956bd4b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:27:45.788403 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.788362 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b62a14-f1f7-4533-b64e-3fda0956bd4b-kube-api-access-gfkhw" (OuterVolumeSpecName: "kube-api-access-gfkhw") pod "89b62a14-f1f7-4533-b64e-3fda0956bd4b" (UID: "89b62a14-f1f7-4533-b64e-3fda0956bd4b"). InnerVolumeSpecName "kube-api-access-gfkhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:27:45.887668 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.887588 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfkhw\" (UniqueName: \"kubernetes.io/projected/89b62a14-f1f7-4533-b64e-3fda0956bd4b-kube-api-access-gfkhw\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:27:45.887668 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:45.887617 2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b62a14-f1f7-4533-b64e-3fda0956bd4b-extensions-socket-volume\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:27:46.130175 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:46.130141 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f048676-a363-4203-8979-7857de7d08b1" path="/var/lib/kubelet/pods/2f048676-a363-4203-8979-7857de7d08b1/volumes" Jul 30 20:27:46.130489 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:46.130476 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b62a14-f1f7-4533-b64e-3fda0956bd4b" path="/var/lib/kubelet/pods/89b62a14-f1f7-4533-b64e-3fda0956bd4b/volumes" Jul 30 20:27:46.130667 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:46.130656 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1734288-6641-4a60-bacc-767b23c44889" path="/var/lib/kubelet/pods/b1734288-6641-4a60-bacc-767b23c44889/volumes" Jul 30 20:27:46.670379 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:46.670343 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:46.674702 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:46.674673 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="89b62a14-f1f7-4533-b64e-3fda0956bd4b" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:47.619844 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:47.619814 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w776m" Jul 30 20:27:47.622161 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:27:47.622136 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-rvxn7\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="89b62a14-f1f7-4533-b64e-3fda0956bd4b" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-rvxn7" Jul 30 20:27:49.020635 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:49.020591 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:49.120144 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:49.120107 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:51.453056 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:51.453017 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:51.568521 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:51.568490 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:56.672532 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:56.672500 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-f8ztj" Jul 30 20:27:56.673034 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:56.672578 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:27:58.428900 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:58.428864 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:58.527387 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:58.527348 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:58.825748 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:58.825652 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:58.928658 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:58.928625 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:59.026989 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:59.026954 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:27:59.089717 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:27:59.089638 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:06.035994 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:06.035957 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:06.180858 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:06.180822 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:10.846091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:10.846053 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:10.945660 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:10.945624 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:13.166752 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.166722 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv"] Jul 30 20:28:13.187654 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.187631 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.195408 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.195034 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c45ln\"" Jul 30 20:28:13.196204 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.196181 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv"] Jul 30 20:28:13.234618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.234576 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:13.317684 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317644 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.317684 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317691 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.317943 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317707 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41aee649-d954-4e90-b0f4-be92c66efaea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.317943 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317731 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41aee649-d954-4e90-b0f4-be92c66efaea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.317943 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317758 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.317943 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317841 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.317943 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317872 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.317943 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317897 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w46l\" (UniqueName: \"kubernetes.io/projected/41aee649-d954-4e90-b0f4-be92c66efaea-kube-api-access-2w46l\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.318243 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.317977 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41aee649-d954-4e90-b0f4-be92c66efaea-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.418952 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.418862 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41aee649-d954-4e90-b0f4-be92c66efaea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.418952 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.418916 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.418952 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.418954 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419220 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.418975 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419220 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419004 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w46l\" (UniqueName: \"kubernetes.io/projected/41aee649-d954-4e90-b0f4-be92c66efaea-kube-api-access-2w46l\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419220 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419044 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41aee649-d954-4e90-b0f4-be92c66efaea-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419220 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419083 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419220 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419117 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419220 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419142 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41aee649-d954-4e90-b0f4-be92c66efaea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419526 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419378 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419580 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419552 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419690 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419670 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41aee649-d954-4e90-b0f4-be92c66efaea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419762 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419725 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.419762 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.419735 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.421618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.421587 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41aee649-d954-4e90-b0f4-be92c66efaea-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.421725 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.421711 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41aee649-d954-4e90-b0f4-be92c66efaea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.429916 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.429890 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w46l\" (UniqueName: \"kubernetes.io/projected/41aee649-d954-4e90-b0f4-be92c66efaea-kube-api-access-2w46l\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.430021 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.429953 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41aee649-d954-4e90-b0f4-be92c66efaea-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-q48mv\" (UID: \"41aee649-d954-4e90-b0f4-be92c66efaea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.456675 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.456641 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:13.498671 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.498631 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:13.542962 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.542917 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:13.639429 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.639400 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv"] Jul 30 20:28:13.641872 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:28:13.641832 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41aee649_d954_4e90_b0f4_be92c66efaea.slice/crio-260abb24b9b54a30a99ecac269f358e11f0aef87c436dbd1c15b46f7ddd405b2 WatchSource:0}: Error finding container 260abb24b9b54a30a99ecac269f358e11f0aef87c436dbd1c15b46f7ddd405b2: Status 404 returned error can't find the container with id 260abb24b9b54a30a99ecac269f358e11f0aef87c436dbd1c15b46f7ddd405b2 Jul 30 20:28:13.643916 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.643885 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:28:13.643992 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.643948 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:28:13.643992 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.643976 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:28:13.794459 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.794426 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" event={"ID":"41aee649-d954-4e90-b0f4-be92c66efaea","Type":"ContainerStarted","Data":"20ccf19d683ce0b6eb0afb098ed771a393993fffc363d517e0af7aaaf8d09f3e"} Jul 30 20:28:13.794623 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.794480 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" event={"ID":"41aee649-d954-4e90-b0f4-be92c66efaea","Type":"ContainerStarted","Data":"260abb24b9b54a30a99ecac269f358e11f0aef87c436dbd1c15b46f7ddd405b2"} Jul 30 20:28:13.821252 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:13.821180 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" podStartSLOduration=0.821157847 podStartE2EDuration="821.157847ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:28:13.820583618 +0000 UTC m=+420.080896145" watchObservedRunningTime="2026-07-30 20:28:13.821157847 +0000 UTC m=+420.081470373" Jul 30 20:28:14.499611 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:14.499571 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:14.504204 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:14.504180 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:14.799445 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:14.799345 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:14.800469 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:14.800442 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:28:14.908093 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:14.908055 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:15.032019 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.031980 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:15.096065 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.095978 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:28:15.100897 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.100871 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.103337 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.103312 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 30 20:28:15.112981 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.112956 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:28:15.132376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.132342 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:15.151930 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.151900 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:28:15.237635 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.237604 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6bc9b063-3816-4a3d-8f80-74d05551a229-config-file\") pod \"limitador-limitador-85d9db4d5d-pgldp\" (UID: \"6bc9b063-3816-4a3d-8f80-74d05551a229\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.237882 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.237647 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/6bc9b063-3816-4a3d-8f80-74d05551a229-kube-api-access-nm7fq\") pod \"limitador-limitador-85d9db4d5d-pgldp\" (UID: \"6bc9b063-3816-4a3d-8f80-74d05551a229\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.338266 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.338234 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/6bc9b063-3816-4a3d-8f80-74d05551a229-kube-api-access-nm7fq\") pod \"limitador-limitador-85d9db4d5d-pgldp\" (UID: \"6bc9b063-3816-4a3d-8f80-74d05551a229\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.338441 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.338321 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6bc9b063-3816-4a3d-8f80-74d05551a229-config-file\") pod \"limitador-limitador-85d9db4d5d-pgldp\" (UID: \"6bc9b063-3816-4a3d-8f80-74d05551a229\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.338959 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.338939 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6bc9b063-3816-4a3d-8f80-74d05551a229-config-file\") pod \"limitador-limitador-85d9db4d5d-pgldp\" (UID: \"6bc9b063-3816-4a3d-8f80-74d05551a229\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.346362 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.346303 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7fq\" (UniqueName: \"kubernetes.io/projected/6bc9b063-3816-4a3d-8f80-74d05551a229-kube-api-access-nm7fq\") pod \"limitador-limitador-85d9db4d5d-pgldp\" (UID: \"6bc9b063-3816-4a3d-8f80-74d05551a229\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.412935 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.412905 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:15.545091 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.545036 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:28:15.547099 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:28:15.547067 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc9b063_3816_4a3d_8f80_74d05551a229.slice/crio-78856967ecc6af642b5d784e67b4bf8a3aabf450f7febd682125bd78220d0040 WatchSource:0}: Error finding container 78856967ecc6af642b5d784e67b4bf8a3aabf450f7febd682125bd78220d0040: Status 404 returned error can't find the container with id 78856967ecc6af642b5d784e67b4bf8a3aabf450f7febd682125bd78220d0040 Jul 30 20:28:15.762537 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.762503 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:15.804198 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.804153 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" event={"ID":"6bc9b063-3816-4a3d-8f80-74d05551a229","Type":"ContainerStarted","Data":"78856967ecc6af642b5d784e67b4bf8a3aabf450f7febd682125bd78220d0040"} Jul 30 20:28:15.882558 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:15.882519 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:17.818467 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:17.818429 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" event={"ID":"6bc9b063-3816-4a3d-8f80-74d05551a229","Type":"ContainerStarted","Data":"8559c581b5d63b32a8080a61b7aded37f97de56e9c7f8b1e32e238e2cf50c4b0"} Jul 30 20:28:17.818946 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:17.818492 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:17.838664 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:17.838609 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" podStartSLOduration=1.3875195790000001 podStartE2EDuration="2.838593982s" totalImagesPullingTime="1.451074403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:28:17.837712401 +0000 UTC m=+424.098024928" watchObservedRunningTime="2026-07-30 20:28:17.838593982 +0000 UTC m=+424.098906517" Jul 30 20:28:28.824492 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:28.824457 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-pgldp" Jul 30 20:28:31.010060 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:31.010012 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:31.109117 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:31.109074 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:31.520770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:31.520730 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:43.679892 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.679800 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:43.776190 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.776150 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:43.912708 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.912667 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4gnq8"] Jul 30 20:28:43.916019 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.915999 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" Jul 30 20:28:43.918407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.918387 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grtts\"" Jul 30 20:28:43.922211 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.922181 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4gnq8"] Jul 30 20:28:43.955905 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.955818 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4gnq8"] Jul 30 20:28:43.956082 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:28:43.956047 2537 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j74km], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" podUID="5561673b-5647-484f-9500-b0dcdd7b25a7" Jul 30 20:28:43.956174 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:43.956151 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/5561673b-5647-484f-9500-b0dcdd7b25a7-kube-api-access-j74km\") pod \"authorino-5dc7ccb5f-4gnq8\" (UID: \"5561673b-5647-484f-9500-b0dcdd7b25a7\") " pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" Jul 30 20:28:44.057162 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:44.057120 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/5561673b-5647-484f-9500-b0dcdd7b25a7-kube-api-access-j74km\") pod \"authorino-5dc7ccb5f-4gnq8\" (UID: \"5561673b-5647-484f-9500-b0dcdd7b25a7\") " pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" Jul 30 20:28:44.066580 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:44.066556 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/5561673b-5647-484f-9500-b0dcdd7b25a7-kube-api-access-j74km\") pod \"authorino-5dc7ccb5f-4gnq8\" (UID: \"5561673b-5647-484f-9500-b0dcdd7b25a7\") " pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" Jul 30 20:28:44.933944 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:44.933906 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" Jul 30 20:28:44.941403 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:44.941379 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" Jul 30 20:28:44.964347 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:44.964316 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/5561673b-5647-484f-9500-b0dcdd7b25a7-kube-api-access-j74km\") pod \"5561673b-5647-484f-9500-b0dcdd7b25a7\" (UID: \"5561673b-5647-484f-9500-b0dcdd7b25a7\") " Jul 30 20:28:44.966368 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:44.966339 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5561673b-5647-484f-9500-b0dcdd7b25a7-kube-api-access-j74km" (OuterVolumeSpecName: "kube-api-access-j74km") pod "5561673b-5647-484f-9500-b0dcdd7b25a7" (UID: "5561673b-5647-484f-9500-b0dcdd7b25a7"). InnerVolumeSpecName "kube-api-access-j74km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:28:45.065736 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:45.065698 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j74km\" (UniqueName: \"kubernetes.io/projected/5561673b-5647-484f-9500-b0dcdd7b25a7-kube-api-access-j74km\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:28:45.938239 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:45.938207 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-4gnq8" Jul 30 20:28:45.967760 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:45.967729 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4gnq8"] Jul 30 20:28:45.970120 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:45.970086 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-4gnq8"] Jul 30 20:28:46.129595 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:46.129557 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5561673b-5647-484f-9500-b0dcdd7b25a7" path="/var/lib/kubelet/pods/5561673b-5647-484f-9500-b0dcdd7b25a7/volumes" Jul 30 20:28:50.917677 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:50.917639 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:51.000893 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:51.000856 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:51.049452 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:51.049414 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:51.279376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:51.279297 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:54.578413 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:54.578367 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:28:54.672203 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:28:54.672165 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:03.186249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:03.186194 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:03.238577 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:03.238543 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:21.668438 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:21.668395 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:21.768055 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:21.768022 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:25.589612 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:25.589581 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:25.656197 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:25.656151 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:27.108081 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:27.108041 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:27.611566 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:27.611531 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:27.712504 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:27.712468 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:27.782255 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:27.782218 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:27.841566 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:27.841531 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:29:37.158810 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:37.158748 2537 generic.go:356] "Generic (PLEG): container finished" podID="41aee649-d954-4e90-b0f4-be92c66efaea" containerID="20ccf19d683ce0b6eb0afb098ed771a393993fffc363d517e0af7aaaf8d09f3e" exitCode=0 Jul 30 20:29:37.159273 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:37.158825 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" event={"ID":"41aee649-d954-4e90-b0f4-be92c66efaea","Type":"ContainerDied","Data":"20ccf19d683ce0b6eb0afb098ed771a393993fffc363d517e0af7aaaf8d09f3e"} Jul 30 20:29:37.159273 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:37.159186 2537 scope.go:117] "RemoveContainer" containerID="20ccf19d683ce0b6eb0afb098ed771a393993fffc363d517e0af7aaaf8d09f3e" Jul 30 20:29:37.160164 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:37.160138 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:29:37.160234 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:37.160221 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:29:37.160270 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:37.160256 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 20:29:38.165160 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:38.165123 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" event={"ID":"41aee649-d954-4e90-b0f4-be92c66efaea","Type":"ContainerStarted","Data":"8570fe7e91fefe80d61a36280df0e4602c11472dca7b36652db85aca3fd828e4"} Jul 30 20:29:38.353630 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:38.353592 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:38.415587 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:38.415487 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:38.499889 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:38.499845 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:29:38.500097 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:38.499931 2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-q48mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 30 20:29:38.500097 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:38.499988 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" podUID="41aee649-d954-4e90-b0f4-be92c66efaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 30 20:29:39.499435 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:39.499399 2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-q48mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 30 20:29:39.500017 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:39.499985 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" podUID="41aee649-d954-4e90-b0f4-be92c66efaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 30 20:29:40.499797 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:40.499734 2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-q48mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 30 20:29:40.500295 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:40.499839 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" podUID="41aee649-d954-4e90-b0f4-be92c66efaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 30 20:29:41.500844 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:41.500798 2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-q48mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 30 20:29:41.501336 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:41.500876 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" podUID="41aee649-d954-4e90-b0f4-be92c66efaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 30 20:29:42.499817 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:42.499280 2537 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-q48mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 30 20:29:42.499817 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:42.499366 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" podUID="41aee649-d954-4e90-b0f4-be92c66efaea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 30 20:29:43.502939 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:43.502908 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:29:43.503352 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:43.503200 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:29:43.504046 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:43.504026 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-q48mv" Jul 30 20:29:57.666049 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:57.666010 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:29:58.056075 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:29:58.055992 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:30:25.759109 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:25.759009 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:30:26.452740 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:26.452703 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:30:30.379695 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:30.379658 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:30:31.557188 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:31.557149 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:30:41.352762 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.352724 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m"] Jul 30 20:30:41.356471 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.356449 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.360028 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.360002 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 30 20:30:41.360028 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.360017 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-67tzk\"" Jul 30 20:30:41.360209 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.360018 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 20:30:41.360209 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.360022 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 20:30:41.366254 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.366229 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m"] Jul 30 20:30:41.388830 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.388736 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:30:41.522534 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.522502 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.522732 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.522590 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca509ce-aa1a-493b-bee7-8e957ef1be28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.522732 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.522646 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.522732 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.522671 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.522919 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.522739 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.522919 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.522772 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2kh\" (UniqueName: \"kubernetes.io/projected/9ca509ce-aa1a-493b-bee7-8e957ef1be28-kube-api-access-dd2kh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.623512 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.623456 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2kh\" (UniqueName: \"kubernetes.io/projected/9ca509ce-aa1a-493b-bee7-8e957ef1be28-kube-api-access-dd2kh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.623512 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.623517 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.623770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.623559 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca509ce-aa1a-493b-bee7-8e957ef1be28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.623770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.623592 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.623770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.623608 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.623770 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.623638 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.624445 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.624063 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.624445 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.624104 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.624445 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.624170 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.626053 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.626031 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca509ce-aa1a-493b-bee7-8e957ef1be28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.626342 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.626320 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca509ce-aa1a-493b-bee7-8e957ef1be28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.631311 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.631291 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2kh\" (UniqueName: \"kubernetes.io/projected/9ca509ce-aa1a-493b-bee7-8e957ef1be28-kube-api-access-dd2kh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m\" (UID: \"9ca509ce-aa1a-493b-bee7-8e957ef1be28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.668347 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.668306 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:41.801407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:41.801382 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m"] Jul 30 20:30:41.804302 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:30:41.804276 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca509ce_aa1a_493b_bee7_8e957ef1be28.slice/crio-c1068d7b85100111d84de5b2c5a7a6e26a11571ae7a17da82b37323f0f4cf018 WatchSource:0}: Error finding container c1068d7b85100111d84de5b2c5a7a6e26a11571ae7a17da82b37323f0f4cf018: Status 404 returned error can't find the container with id c1068d7b85100111d84de5b2c5a7a6e26a11571ae7a17da82b37323f0f4cf018 Jul 30 20:30:42.443193 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:42.443158 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" event={"ID":"9ca509ce-aa1a-493b-bee7-8e957ef1be28","Type":"ContainerStarted","Data":"c1068d7b85100111d84de5b2c5a7a6e26a11571ae7a17da82b37323f0f4cf018"} Jul 30 20:30:42.660192 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:42.660154 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:30:43.449614 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:43.449566 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" event={"ID":"9ca509ce-aa1a-493b-bee7-8e957ef1be28","Type":"ContainerStarted","Data":"f25d5dd0aedaaa9053484e9f7e018b4e1142a1d283f2dcc4da3241ce482c45ca"} Jul 30 20:30:43.471038 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:43.470982 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" podStartSLOduration=1.468362561 podStartE2EDuration="2.470964035s" totalImagesPullingTime="1.002601474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:30:43.468469288 +0000 UTC m=+569.728781816" watchObservedRunningTime="2026-07-30 20:30:43.470964035 +0000 UTC m=+569.731276560" Jul 30 20:30:51.669189 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:51.669140 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:51.669637 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:51.669207 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:51.681837 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:51.681809 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:30:52.500064 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:30:52.500033 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m" Jul 30 20:31:06.265222 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.265179 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n"] Jul 30 20:31:06.268593 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.268576 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.271414 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.271393 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 30 20:31:06.278512 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.278488 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n"] Jul 30 20:31:06.305344 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.305271 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:31:06.320423 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.320386 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzrn\" (UniqueName: \"kubernetes.io/projected/55d84486-6ada-49fc-99ed-ef126e46cd75-kube-api-access-zvzrn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.320649 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.320437 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.320649 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.320484 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.320649 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.320523 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.320649 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.320546 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d84486-6ada-49fc-99ed-ef126e46cd75-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.320649 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.320583 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421400 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421364 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzrn\" (UniqueName: \"kubernetes.io/projected/55d84486-6ada-49fc-99ed-ef126e46cd75-kube-api-access-zvzrn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421550 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421408 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421550 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421434 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421550 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421463 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421550 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421484 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d84486-6ada-49fc-99ed-ef126e46cd75-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421550 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421517 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421867 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421848 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421930 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421879 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.421981 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.421933 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.423745 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.423719 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d84486-6ada-49fc-99ed-ef126e46cd75-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.424061 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.424046 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d84486-6ada-49fc-99ed-ef126e46cd75-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.431206 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.431181 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzrn\" (UniqueName: \"kubernetes.io/projected/55d84486-6ada-49fc-99ed-ef126e46cd75-kube-api-access-zvzrn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-96d6n\" (UID: \"55d84486-6ada-49fc-99ed-ef126e46cd75\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.580099 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.579990 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:06.713572 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:06.713537 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n"] Jul 30 20:31:06.716858 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:31:06.716832 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d84486_6ada_49fc_99ed_ef126e46cd75.slice/crio-bac579ef75e90ba7f1030be324dc4c2fe7bb2324eb0443c7bdcf7cfe9329bdfd WatchSource:0}: Error finding container bac579ef75e90ba7f1030be324dc4c2fe7bb2324eb0443c7bdcf7cfe9329bdfd: Status 404 returned error can't find the container with id bac579ef75e90ba7f1030be324dc4c2fe7bb2324eb0443c7bdcf7cfe9329bdfd Jul 30 20:31:07.558508 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:07.558470 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" event={"ID":"55d84486-6ada-49fc-99ed-ef126e46cd75","Type":"ContainerStarted","Data":"9837b4ae3fdfb6aa4319dc748bd7891d316264380bb2357112d321de6e4eb2cb"} Jul 30 20:31:07.558963 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:07.558526 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" event={"ID":"55d84486-6ada-49fc-99ed-ef126e46cd75","Type":"ContainerStarted","Data":"bac579ef75e90ba7f1030be324dc4c2fe7bb2324eb0443c7bdcf7cfe9329bdfd"} Jul 30 20:31:07.576682 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:07.576633 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" podStartSLOduration=1.386120654 podStartE2EDuration="1.576616221s" totalImagesPullingTime="190.495567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:31:07.574622375 +0000 UTC m=+593.834934901" watchObservedRunningTime="2026-07-30 20:31:07.576616221 +0000 UTC m=+593.836928748" Jul 30 20:31:08.036918 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:08.036883 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:31:08.454199 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:08.454163 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:31:14.249754 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.249727 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf"] Jul 30 20:31:14.254914 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.254889 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.257484 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.257463 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 30 20:31:14.261692 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.261643 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf"] Jul 30 20:31:14.278260 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.278230 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:31:14.288407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.288379 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.288589 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.288416 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.288589 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.288444 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c748b4-fc2e-4669-8546-3ed8122e5cf9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.288589 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.288482 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.288589 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.288526 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.288795 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.288610 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47hd\" (UniqueName: \"kubernetes.io/projected/07c748b4-fc2e-4669-8546-3ed8122e5cf9-kube-api-access-p47hd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.389618 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.389558 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47hd\" (UniqueName: \"kubernetes.io/projected/07c748b4-fc2e-4669-8546-3ed8122e5cf9-kube-api-access-p47hd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.389846 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.389639 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.389846 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.389658 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.389846 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.389677 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c748b4-fc2e-4669-8546-3ed8122e5cf9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.389846 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.389702 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.389846 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.389726 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.390127 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.390105 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.390204 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.390183 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.390240 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.390207 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.392016 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.391998 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c748b4-fc2e-4669-8546-3ed8122e5cf9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.392296 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.392274 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c748b4-fc2e-4669-8546-3ed8122e5cf9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.397264 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.397240 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47hd\" (UniqueName: \"kubernetes.io/projected/07c748b4-fc2e-4669-8546-3ed8122e5cf9-kube-api-access-p47hd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gppwf\" (UID: \"07c748b4-fc2e-4669-8546-3ed8122e5cf9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.565376 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.565285 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:14.703895 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:14.703854 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf"] Jul 30 20:31:14.706979 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:31:14.706949 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07c748b4_fc2e_4669_8546_3ed8122e5cf9.slice/crio-2432333d73b273d4e70d5f4dee75b924d333c6c428b39a9c17492596c90f6f27 WatchSource:0}: Error finding container 2432333d73b273d4e70d5f4dee75b924d333c6c428b39a9c17492596c90f6f27: Status 404 returned error can't find the container with id 2432333d73b273d4e70d5f4dee75b924d333c6c428b39a9c17492596c90f6f27 Jul 30 20:31:15.595878 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:15.595844 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" event={"ID":"07c748b4-fc2e-4669-8546-3ed8122e5cf9","Type":"ContainerStarted","Data":"2432333d73b273d4e70d5f4dee75b924d333c6c428b39a9c17492596c90f6f27"} Jul 30 20:31:16.580128 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:16.580091 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:16.580398 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:16.580376 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:16.592915 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:16.592891 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:16.601406 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:16.601370 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" event={"ID":"07c748b4-fc2e-4669-8546-3ed8122e5cf9","Type":"ContainerStarted","Data":"dd59723dc03c41f2868280f029f750657b51822aaad55ed23136353f08896eb5"} Jul 30 20:31:16.613261 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:16.613233 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-96d6n" Jul 30 20:31:16.623993 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:16.623946 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" podStartSLOduration=1.385468122 podStartE2EDuration="2.6239303s" totalImagesPullingTime="1.238462178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:31:16.623241803 +0000 UTC m=+602.883554331" watchObservedRunningTime="2026-07-30 20:31:16.6239303 +0000 UTC m=+602.884242804" Jul 30 20:31:16.854682 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:16.854603 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:31:24.566246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:24.566203 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:24.566246 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:24.566256 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:24.578873 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:24.578840 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:24.649353 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:24.649326 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gppwf" Jul 30 20:31:33.309825 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:31:33.309771 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:32:18.942123 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:32:18.942073 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:33:34.118985 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:33:34.118889 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:33:34.220975 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:33:34.220938 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:33:35.260951 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:33:35.260915 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:33:35.361373 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:33:35.361337 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:33:52.156195 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:33:52.156156 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:33:52.207651 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:33:52.207618 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:34:02.852251 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:34:02.852213 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:34:12.656281 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:34:12.656244 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:34:23.961217 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:34:23.961175 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:35:04.513129 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:35:04.513085 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:35:36.601275 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:35:36.601238 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:35:40.990879 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:35:40.990839 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:35:42.266894 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:35:42.266858 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:36:06.364209 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:06.364116 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:36:14.354821 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:14.354769 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:36:53.103300 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:53.103255 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:36:54.460212 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:54.460169 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:36:59.317846 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:59.317807 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:36:59.386280 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:59.386231 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:36:59.464027 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:59.463989 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:36:59.563764 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:59.563726 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:36:59.682237 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:36:59.682195 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:37:05.214564 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:37:05.214527 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:37:05.271119 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:37:05.271085 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:37:20.952136 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:37:20.952094 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:37:28.250934 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:37:28.250899 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:37:43.655475 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:37:43.655394 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:37:49.755642 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:37:49.755605 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:37:59.155443 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:37:59.155407 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:38:05.852284 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:38:05.852248 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:38:19.755028 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:38:19.754987 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:38:24.852879 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:38:24.852838 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:38:30.258231 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:38:30.258195 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:38:51.951167 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:38:51.951130 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:39:02.464567 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:39:02.464530 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:39:25.161459 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:39:25.161369 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:39:35.554879 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:39:35.554833 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:39:40.749415 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:39:40.749378 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:39:48.154342 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:39:48.154299 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:39:55.694789 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:39:55.694749 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:01.555809 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:01.555761 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:07.162651 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:07.162614 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:14.554556 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:14.554518 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:18.711238 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:18.711197 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:40:26.258263 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:26.258229 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:33.657045 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:33.657006 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:44.248430 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:44.248345 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:49.262070 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:49.262033 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:40:55.654234 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:40:55.654199 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:41:04.850217 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:41:04.850182 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:41:09.961002 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:41:09.960965 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:41:20.855271 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:41:20.855231 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:41:35.332690 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:41:35.332649 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:41:35.433927 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:41:35.433885 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:42:16.122576 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.122483 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc"] Jul 30 20:42:16.123182 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.122748 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" podUID="4a701e93-3bae-4371-9349-b6747dacb652" containerName="manager" containerID="cri-o://0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40" gracePeriod=10 Jul 30 20:42:16.371248 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.371215 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:42:16.429288 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.429205 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a701e93-3bae-4371-9349-b6747dacb652-extensions-socket-volume\") pod \"4a701e93-3bae-4371-9349-b6747dacb652\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " Jul 30 20:42:16.429288 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.429270 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nq77\" (UniqueName: \"kubernetes.io/projected/4a701e93-3bae-4371-9349-b6747dacb652-kube-api-access-7nq77\") pod \"4a701e93-3bae-4371-9349-b6747dacb652\" (UID: \"4a701e93-3bae-4371-9349-b6747dacb652\") " Jul 30 20:42:16.429687 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.429659 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a701e93-3bae-4371-9349-b6747dacb652-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4a701e93-3bae-4371-9349-b6747dacb652" (UID: "4a701e93-3bae-4371-9349-b6747dacb652"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:42:16.431328 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.431300 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a701e93-3bae-4371-9349-b6747dacb652-kube-api-access-7nq77" (OuterVolumeSpecName: "kube-api-access-7nq77") pod "4a701e93-3bae-4371-9349-b6747dacb652" (UID: "4a701e93-3bae-4371-9349-b6747dacb652"). InnerVolumeSpecName "kube-api-access-7nq77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:42:16.530472 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.530429 2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a701e93-3bae-4371-9349-b6747dacb652-extensions-socket-volume\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:42:16.530472 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.530466 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nq77\" (UniqueName: \"kubernetes.io/projected/4a701e93-3bae-4371-9349-b6747dacb652-kube-api-access-7nq77\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:42:16.614236 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.614202 2537 generic.go:356] "Generic (PLEG): container finished" podID="4a701e93-3bae-4371-9349-b6747dacb652" containerID="0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40" exitCode=0 Jul 30 20:42:16.614424 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.614253 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" Jul 30 20:42:16.614424 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.614283 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" event={"ID":"4a701e93-3bae-4371-9349-b6747dacb652","Type":"ContainerDied","Data":"0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40"} Jul 30 20:42:16.614424 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.614345 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc" event={"ID":"4a701e93-3bae-4371-9349-b6747dacb652","Type":"ContainerDied","Data":"690989bd436102d5bce8078285282f69f1ad1fc8458802e797ce2ac5bbc03b86"} Jul 30 20:42:16.614424 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.614374 2537 scope.go:117] "RemoveContainer" containerID="0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40" Jul 30 20:42:16.624316 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.624301 2537 scope.go:117] "RemoveContainer" containerID="0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40" Jul 30 20:42:16.624580 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:42:16.624554 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40\": container with ID starting with 0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40 not found: ID does not exist" containerID="0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40" Jul 30 20:42:16.624668 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.624587 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40"} err="failed to get container status \"0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40\": rpc error: code = NotFound desc = could not find container \"0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40\": container with ID starting with 0f8f16f99795211fc1aa649b415620d17f4143b8ae649578d3b6c315af6cfa40 not found: ID does not exist" Jul 30 20:42:16.636377 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.636355 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc"] Jul 30 20:42:16.639644 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:16.639626 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-c5slc"] Jul 30 20:42:18.129440 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:18.129407 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a701e93-3bae-4371-9349-b6747dacb652" path="/var/lib/kubelet/pods/4a701e93-3bae-4371-9349-b6747dacb652/volumes" Jul 30 20:42:31.356650 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:31.356611 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:42:46.614575 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:42:46.614542 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:43:16.849614 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.849577 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275"] Jul 30 20:43:16.850043 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.849916 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a701e93-3bae-4371-9349-b6747dacb652" containerName="manager" Jul 30 20:43:16.850043 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.849928 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a701e93-3bae-4371-9349-b6747dacb652" containerName="manager" Jul 30 20:43:16.850043 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.849992 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a701e93-3bae-4371-9349-b6747dacb652" containerName="manager" Jul 30 20:43:16.853307 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.853290 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:16.856026 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.856007 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56ccm\"" Jul 30 20:43:16.864965 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.864931 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275"] Jul 30 20:43:16.969734 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.969688 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5596b\" (UniqueName: \"kubernetes.io/projected/6cbbc915-7e54-4411-973a-6170b6252549-kube-api-access-5596b\") pod \"kuadrant-operator-controller-manager-68cf589589-js275\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:16.969734 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:16.969738 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbbc915-7e54-4411-973a-6170b6252549-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-js275\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:17.070417 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.070372 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5596b\" (UniqueName: \"kubernetes.io/projected/6cbbc915-7e54-4411-973a-6170b6252549-kube-api-access-5596b\") pod \"kuadrant-operator-controller-manager-68cf589589-js275\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:17.070417 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.070417 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbbc915-7e54-4411-973a-6170b6252549-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-js275\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:17.070800 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.070756 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbbc915-7e54-4411-973a-6170b6252549-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-js275\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:17.085216 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.085184 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5596b\" (UniqueName: \"kubernetes.io/projected/6cbbc915-7e54-4411-973a-6170b6252549-kube-api-access-5596b\") pod \"kuadrant-operator-controller-manager-68cf589589-js275\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:17.163150 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.163106 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:17.321791 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.321691 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275"] Jul 30 20:43:17.324355 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:43:17.324322 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbbc915_7e54_4411_973a_6170b6252549.slice/crio-9f674b9d4a64c773d1e0179e7d537c42f51e9d46d4e734f5b1663089cffb06ba WatchSource:0}: Error finding container 9f674b9d4a64c773d1e0179e7d537c42f51e9d46d4e734f5b1663089cffb06ba: Status 404 returned error can't find the container with id 9f674b9d4a64c773d1e0179e7d537c42f51e9d46d4e734f5b1663089cffb06ba Jul 30 20:43:17.887235 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.887200 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" event={"ID":"6cbbc915-7e54-4411-973a-6170b6252549","Type":"ContainerStarted","Data":"85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055"} Jul 30 20:43:17.887631 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.887257 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" event={"ID":"6cbbc915-7e54-4411-973a-6170b6252549","Type":"ContainerStarted","Data":"9f674b9d4a64c773d1e0179e7d537c42f51e9d46d4e734f5b1663089cffb06ba"} Jul 30 20:43:17.887631 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.887284 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:17.908834 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:17.908763 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" podStartSLOduration=1.908748345 podStartE2EDuration="1.908748345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:43:17.906951006 +0000 UTC m=+1324.167263532" watchObservedRunningTime="2026-07-30 20:43:17.908748345 +0000 UTC m=+1324.169060871" Jul 30 20:43:28.894212 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:28.894181 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:43:36.051939 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:36.051851 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:43:41.061823 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:43:41.061789 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:44:07.162044 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:44:07.162006 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:44:16.556039 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:44:16.556004 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:44:23.752081 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:44:23.752044 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:44:31.549472 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:44:31.549435 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:44:37.157149 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:44:37.157114 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:44:47.159568 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:44:47.159527 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:44:54.560156 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:44:54.560121 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:45:02.449869 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:45:02.449828 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:45:08.054491 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:45:08.054410 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:45:17.557750 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:45:17.557716 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:45:23.757902 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:45:23.757868 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:45:32.754474 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:45:32.754437 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:45:55.796865 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:45:55.796829 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:45:58.161704 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:45:58.161668 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:46:03.553146 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:03.553109 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:46:07.393075 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:07.393030 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:46:09.750120 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:09.750082 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:46:18.006211 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:18.006169 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:46:21.259821 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:21.259768 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:46:47.724534 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:47.724445 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:46:49.404225 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:49.404180 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:46:49.856129 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:46:49.856037 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:47:01.149628 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:01.149586 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:47:08.860357 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:08.860317 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:47:17.360768 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:17.360733 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:47:22.353685 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:22.353646 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:47:29.013925 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:29.013884 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:47:35.863071 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:35.863035 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:47:41.955423 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:41.955384 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:47:49.736826 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:49.736767 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:47:54.760218 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:47:54.760175 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:00.755027 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:00.754982 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:17.365665 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:17.365587 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:22.354384 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:22.354349 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:35.758895 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:35.758861 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:41.161397 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:41.161352 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:46.959651 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:46.959611 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:51.856949 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:51.856915 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:48:59.258901 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:48:59.258863 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:49:08.559790 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:08.559738 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:49:15.761889 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:15.761844 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:49:25.561574 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:25.561540 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:49:32.563810 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:32.563757 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:49:40.380075 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:40.379988 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:49:40.431047 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:40.431017 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:49:42.757428 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:42.757390 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:49:43.659797 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:43.659738 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:49:57.953046 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:49:57.953006 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:50:04.051598 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:04.051565 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:50:07.684797 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:07.684752 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:10.458396 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:10.458359 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:50:22.384507 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:22.384472 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:50:30.277843 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:30.277806 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:50:33.716726 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:33.716692 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:35.762549 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:35.762512 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:50:36.930540 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:36.930508 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:37.022629 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:37.022597 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:37.391481 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:37.391445 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:43.253010 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:43.252975 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:43.353033 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:43.353004 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:43.541103 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:43.541024 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:43.916435 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:43.916395 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:44.136243 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:44.136214 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:49.157586 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:49.157546 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:49.258678 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:49.258643 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:50.301327 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:50.301296 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:50.420921 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:50.420884 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:50.886666 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:50.886626 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:50.976093 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:50.976055 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:50:52.768289 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:50:52.768256 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:51:20.822333 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:20.822299 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:20.927583 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:20.927541 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:21.029763 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:21.029727 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:21.110550 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:21.110520 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:21.214687 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:21.214651 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:21.440615 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:21.440531 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:21.708685 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:21.708608 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:21.807547 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:21.807508 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:21.927441 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:21.927410 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:22.293449 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:22.293414 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:22.375485 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:22.375439 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:23.576443 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:23.576411 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:51:24.509039 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:24.508999 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:24.663890 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:24.663863 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:25.213144 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:25.213091 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:25.315268 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:25.315238 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:25.475864 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:25.475738 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:31.074243 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:31.074193 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:51:56.170965 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:56.170927 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:56.276696 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:56.276666 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:56.375231 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:56.375196 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:56.469389 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:56.469308 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:56.673546 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:56.673506 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:56.923792 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:56.923756 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:57.187426 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:57.187332 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:57.493035 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:57.492953 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:57.776276 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:57.776197 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:59.161436 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:59.161404 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:51:59.647818 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:59.647738 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:51:59.744904 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:51:59.744875 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:00.756604 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:00.756566 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:00.978166 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:00.978130 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:01.396576 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:01.396520 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:01.492063 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:01.492021 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:07.074153 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:07.074113 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:52:31.320446 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:31.320416 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:31.396704 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:31.396672 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:31.464065 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:31.464032 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:31.562624 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:31.562586 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:32.739939 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:32.739903 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:32.959609 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:32.959576 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:33.184372 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:33.184333 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:33.566042 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:33.565933 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:33.681321 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:33.681278 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:34.457274 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:34.457233 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:52:35.704365 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:35.704318 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:35.800190 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:35.800140 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:36.935602 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:36.935570 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:37.418889 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:37.418854 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:37.500855 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:37.500817 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:52:42.252510 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:52:42.252473 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:53:07.433841 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:07.433762 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:08.682099 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:08.682063 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:08.784099 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:08.784065 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:08.881507 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:08.881470 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:09.553261 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:09.553227 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:53:16.552865 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:16.552831 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:53:20.623498 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:20.623459 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:23.556092 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:23.556052 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:53:27.058226 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:27.058186 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:27.162016 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:27.161982 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:27.253109 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:27.253074 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:27.354114 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:27.354041 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:27.859007 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:27.858976 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:29.512974 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:29.512935 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:29.613863 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:29.613833 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:53:53.843394 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:53:53.843358 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:00.134788 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:00.134744 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:00.226967 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:00.226931 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:00.326145 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:00.326109 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:16.238402 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:16.238318 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:21.834529 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:21.834492 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:21.939283 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:21.939252 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:22.037274 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:22.037246 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:32.568263 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:32.568225 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:41.573370 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:41.573332 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:43.596080 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:43.596051 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:49.456829 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:49.456791 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:49.566690 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:49.566648 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:49.694592 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:49.694557 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:49.814277 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:49.814185 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:50.428116 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:50.428085 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:50.524006 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:50.523976 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:50.625182 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:50.625137 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:52.119838 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:52.119804 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:56.967381 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:56.967338 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:54:58.301227 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:58.301192 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:58.400315 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:58.400277 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:54:58.984553 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:54:58.984516 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:01.050504 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:01.050460 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:55:05.063158 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:05.063122 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:05.164606 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:05.164562 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:05.322688 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:05.322601 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:05.725244 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:05.725206 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:05.977809 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:05.977705 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:08.151936 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:08.151894 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:09.185297 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:09.185266 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:09.334584 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:09.334554 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:10.343049 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:10.343016 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:10.852435 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:10.852404 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:10.947289 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:10.947253 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:12.752628 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:12.752593 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:55:34.129090 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:34.129045 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:34.805109 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:34.805072 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:34.903005 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:34.902973 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:35.229895 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:35.229863 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:41.120298 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:41.120217 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:41.220067 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:41.220032 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:52.033224 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:52.033189 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:52.160149 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:52.160117 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:52.248678 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:52.248647 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:52.323049 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:52.322749 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:52.686332 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:52.686298 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:54.451931 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:54.451895 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:55:58.675357 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:58.675321 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:58.775641 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:58.775582 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:58.935838 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:58.935743 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:59.829499 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:59.829461 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:55:59.891204 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:55:59.891175 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:56:05.390130 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:05.390071 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:56:05.493905 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:05.493868 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:56:05.587063 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:05.587032 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:56:05.730747 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:05.730671 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:56:11.377929 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:11.377896 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:56:11.444092 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:11.444064 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:56:32.951646 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:32.951610 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:56:39.830737 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:39.830699 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275"] Jul 30 20:56:39.831313 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:39.830938 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" podUID="6cbbc915-7e54-4411-973a-6170b6252549" containerName="manager" containerID="cri-o://85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055" gracePeriod=10 Jul 30 20:56:40.086266 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.086206 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:56:40.186915 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.186883 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbbc915-7e54-4411-973a-6170b6252549-extensions-socket-volume\") pod \"6cbbc915-7e54-4411-973a-6170b6252549\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " Jul 30 20:56:40.187087 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.186928 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5596b\" (UniqueName: \"kubernetes.io/projected/6cbbc915-7e54-4411-973a-6170b6252549-kube-api-access-5596b\") pod \"6cbbc915-7e54-4411-973a-6170b6252549\" (UID: \"6cbbc915-7e54-4411-973a-6170b6252549\") " Jul 30 20:56:40.187378 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.187345 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbbc915-7e54-4411-973a-6170b6252549-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6cbbc915-7e54-4411-973a-6170b6252549" (UID: "6cbbc915-7e54-4411-973a-6170b6252549"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 20:56:40.189028 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.188998 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbbc915-7e54-4411-973a-6170b6252549-kube-api-access-5596b" (OuterVolumeSpecName: "kube-api-access-5596b") pod "6cbbc915-7e54-4411-973a-6170b6252549" (UID: "6cbbc915-7e54-4411-973a-6170b6252549"). InnerVolumeSpecName "kube-api-access-5596b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 20:56:40.287751 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.287716 2537 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cbbc915-7e54-4411-973a-6170b6252549-extensions-socket-volume\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:56:40.287751 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.287746 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5596b\" (UniqueName: \"kubernetes.io/projected/6cbbc915-7e54-4411-973a-6170b6252549-kube-api-access-5596b\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 20:56:40.576594 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.576557 2537 generic.go:356] "Generic (PLEG): container finished" podID="6cbbc915-7e54-4411-973a-6170b6252549" containerID="85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055" exitCode=0 Jul 30 20:56:40.576765 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.576610 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" Jul 30 20:56:40.576765 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.576633 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" event={"ID":"6cbbc915-7e54-4411-973a-6170b6252549","Type":"ContainerDied","Data":"85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055"} Jul 30 20:56:40.576765 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.576693 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275" event={"ID":"6cbbc915-7e54-4411-973a-6170b6252549","Type":"ContainerDied","Data":"9f674b9d4a64c773d1e0179e7d537c42f51e9d46d4e734f5b1663089cffb06ba"} Jul 30 20:56:40.576765 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.576722 2537 scope.go:117] "RemoveContainer" containerID="85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055" Jul 30 20:56:40.586709 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.586693 2537 scope.go:117] "RemoveContainer" containerID="85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055" Jul 30 20:56:40.586986 ip-10-0-131-223 kubenswrapper[2537]: E0730 20:56:40.586961 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055\": container with ID starting with 85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055 not found: ID does not exist" containerID="85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055" Jul 30 20:56:40.587075 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.586990 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055"} err="failed to get container status \"85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055\": rpc error: code = NotFound desc = could not find container \"85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055\": container with ID starting with 85aa960ee051980d430868a0a1c72ea8bffc444d9ba71ccec15e18c8ebfc6055 not found: ID does not exist" Jul 30 20:56:40.598105 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.598082 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275"] Jul 30 20:56:40.601129 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:40.601109 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-js275"] Jul 30 20:56:42.129789 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:42.129741 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cbbc915-7e54-4411-973a-6170b6252549" path="/var/lib/kubelet/pods/6cbbc915-7e54-4411-973a-6170b6252549/volumes" Jul 30 20:56:54.250216 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:56:54.250185 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:57:55.457744 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.457693 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw"] Jul 30 20:57:55.458233 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.458050 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cbbc915-7e54-4411-973a-6170b6252549" containerName="manager" Jul 30 20:57:55.458233 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.458062 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cbbc915-7e54-4411-973a-6170b6252549" containerName="manager" Jul 30 20:57:55.458233 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.458133 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cbbc915-7e54-4411-973a-6170b6252549" containerName="manager" Jul 30 20:57:55.460975 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.460959 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.463613 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.463585 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56ccm\"" Jul 30 20:57:55.472610 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.472586 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw"] Jul 30 20:57:55.513952 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.513895 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs54f\" (UniqueName: \"kubernetes.io/projected/2958d992-efcc-4fd7-b0d3-5676d9cb6fac-kube-api-access-cs54f\") pod \"kuadrant-operator-controller-manager-68cf589589-h5bbw\" (UID: \"2958d992-efcc-4fd7-b0d3-5676d9cb6fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.514126 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.514024 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2958d992-efcc-4fd7-b0d3-5676d9cb6fac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-h5bbw\" (UID: \"2958d992-efcc-4fd7-b0d3-5676d9cb6fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.614966 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.614926 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2958d992-efcc-4fd7-b0d3-5676d9cb6fac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-h5bbw\" (UID: \"2958d992-efcc-4fd7-b0d3-5676d9cb6fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.615140 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.614993 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs54f\" (UniqueName: \"kubernetes.io/projected/2958d992-efcc-4fd7-b0d3-5676d9cb6fac-kube-api-access-cs54f\") pod \"kuadrant-operator-controller-manager-68cf589589-h5bbw\" (UID: \"2958d992-efcc-4fd7-b0d3-5676d9cb6fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.615340 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.615319 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2958d992-efcc-4fd7-b0d3-5676d9cb6fac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-h5bbw\" (UID: \"2958d992-efcc-4fd7-b0d3-5676d9cb6fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.622864 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.622837 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs54f\" (UniqueName: \"kubernetes.io/projected/2958d992-efcc-4fd7-b0d3-5676d9cb6fac-kube-api-access-cs54f\") pod \"kuadrant-operator-controller-manager-68cf589589-h5bbw\" (UID: \"2958d992-efcc-4fd7-b0d3-5676d9cb6fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.771285 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.771190 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:55.912313 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:55.912284 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw"] Jul 30 20:57:55.914906 ip-10-0-131-223 kubenswrapper[2537]: W0730 20:57:55.914877 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2958d992_efcc_4fd7_b0d3_5676d9cb6fac.slice/crio-19e3492b7ccabb4f852dd645b2e94b6058db6b8fb66fe83e656f7fbb02116d19 WatchSource:0}: Error finding container 19e3492b7ccabb4f852dd645b2e94b6058db6b8fb66fe83e656f7fbb02116d19: Status 404 returned error can't find the container with id 19e3492b7ccabb4f852dd645b2e94b6058db6b8fb66fe83e656f7fbb02116d19 Jul 30 20:57:56.925407 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:56.925358 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" event={"ID":"2958d992-efcc-4fd7-b0d3-5676d9cb6fac","Type":"ContainerStarted","Data":"f6b7d5f0d1eea544d8e7b433aae4376d25e92e48e8dd92e1d274eecfd8fce5a7"} Jul 30 20:57:56.925820 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:56.925423 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" event={"ID":"2958d992-efcc-4fd7-b0d3-5676d9cb6fac","Type":"ContainerStarted","Data":"19e3492b7ccabb4f852dd645b2e94b6058db6b8fb66fe83e656f7fbb02116d19"} Jul 30 20:57:56.925820 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:56.925536 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:57:56.948622 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:57:56.948564 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" podStartSLOduration=1.948543044 podStartE2EDuration="1.948543044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:56.946882408 +0000 UTC m=+2203.207194934" watchObservedRunningTime="2026-07-30 20:57:56.948543044 +0000 UTC m=+2203.208855576" Jul 30 20:58:04.436362 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:04.436323 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:07.932249 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:07.932218 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-h5bbw" Jul 30 20:58:14.491108 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:14.491072 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:14.814636 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:14.814557 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:58:15.791626 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:15.791592 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:18.067741 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:18.067707 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:18.166047 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:18.166005 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:19.227599 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:19.227566 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:19.329958 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:19.329901 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:19.819085 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:19.819051 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:19.925139 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:19.925099 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:22.053156 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:22.053124 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:58:43.060431 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:43.060347 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:53.583113 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:53.583080 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:55.665323 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:55.665279 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:58:59.242163 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:59.242125 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:58:59.343196 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:58:59.343160 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:59:12.264610 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:12.264573 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:59:35.662339 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:35.662300 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:59:38.136858 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:38.136819 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:59:38.208148 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:38.208115 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:59:38.270533 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:38.270494 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:59:38.371018 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:38.370985 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:59:41.454272 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:41.454234 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 20:59:41.545702 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:41.545667 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 20:59:46.580015 ip-10-0-131-223 kubenswrapper[2537]: I0730 20:59:46.579976 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:23.048741 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:23.048657 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:23.882473 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:23.882422 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:29.551837 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:29.551799 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:29.652326 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:29.652279 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:29.814162 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:29.814064 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:30.056724 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:30.056695 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:31.669948 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:31.669912 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:00:41.526907 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:41.526872 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:41.626829 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:41.626800 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:42.306099 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:42.306069 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:42.408413 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:42.408366 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:43.718562 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:43.718510 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:43.766641 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:43.766609 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:00:50.266377 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:50.266337 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:00:50.475005 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:50.474973 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:50.574984 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:50.574882 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:50.919957 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:50.919920 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:56.615660 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:56.615623 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:56.715162 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:56.715116 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:56.810859 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:56.810814 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:00:59.058873 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:00:59.058833 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:01:07.335982 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:07.335949 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:07.575175 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:07.575136 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:28.572824 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:28.572759 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:34.516458 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:34.516419 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:34.618825 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:34.618788 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:34.880146 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:34.880108 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:35.192196 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:35.192092 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:35.335638 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:35.335604 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:38.133495 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:38.133400 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:38.230354 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:38.230299 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:39.425807 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:39.425738 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:39.763940 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:39.763804 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:40.188529 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:40.188498 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:40.278411 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:40.278368 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:01:42.163360 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:01:42.163325 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:02:03.162494 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:03.162460 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:03.498174 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:03.498059 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:03.834380 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:03.834295 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:03.902330 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:03.902298 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:06.296545 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:06.296510 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:06.395485 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:06.395447 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:07.529280 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:07.529248 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:07.931433 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:07.931378 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:08.035061 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:08.035020 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:08.123242 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:08.123201 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:10.055747 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:10.055710 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:02:31.317245 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:31.317206 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:32.659120 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.659088 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5"] Jul 30 21:02:32.662328 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.662306 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.665402 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.665378 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72f733\"/\"kube-root-ca.crt\"" Jul 30 21:02:32.665402 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.665393 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-72f733\"/\"openshift-service-ca.crt\"" Jul 30 21:02:32.666394 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.666378 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72f733\"/\"test-model-72f733-kserve-self-signed-certs\"" Jul 30 21:02:32.666709 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.666690 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-72f733\"/\"default-dockercfg-7wptp\"" Jul 30 21:02:32.673766 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.673742 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5"] Jul 30 21:02:32.705389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.705354 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:32.747589 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.747543 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwllr\" (UniqueName: \"kubernetes.io/projected/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-kube-api-access-mwllr\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.747806 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.747672 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-home\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.747806 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.747702 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tmp-dir\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.747806 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.747737 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-dshm\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.747806 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.747766 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tls-certs\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.747982 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.747853 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-model-cache\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.848769 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.848722 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-home\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.848769 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.848766 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tmp-dir\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.849044 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.848818 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-dshm\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.849044 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.848840 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tls-certs\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.849044 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.848873 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-model-cache\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.849044 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.848934 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwllr\" (UniqueName: \"kubernetes.io/projected/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-kube-api-access-mwllr\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.849254 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.849162 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-home\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.849254 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.849228 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tmp-dir\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.849350 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.849326 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-model-cache\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.851143 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.851117 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-dshm\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.851441 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.851424 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tls-certs\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.860685 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.860643 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwllr\" (UniqueName: \"kubernetes.io/projected/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-kube-api-access-mwllr\") pod \"test-model-72f733-kserve-5f8459fbf7-wfjp5\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:32.973478 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:32.973376 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:33.120298 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:33.120225 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5"] Jul 30 21:02:33.122456 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:02:33.122425 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4afb3434_4e4c_45e3_8ed8_a3b24d51b47b.slice/crio-826429cf1c4c5c22d30f86c1a1c546fc080d70880f49703c074009156f5f0ffd WatchSource:0}: Error finding container 826429cf1c4c5c22d30f86c1a1c546fc080d70880f49703c074009156f5f0ffd: Status 404 returned error can't find the container with id 826429cf1c4c5c22d30f86c1a1c546fc080d70880f49703c074009156f5f0ffd Jul 30 21:02:33.182599 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:33.182570 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" event={"ID":"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b","Type":"ContainerStarted","Data":"607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef"} Jul 30 21:02:33.182709 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:33.182611 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" event={"ID":"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b","Type":"ContainerStarted","Data":"826429cf1c4c5c22d30f86c1a1c546fc080d70880f49703c074009156f5f0ffd"} Jul 30 21:02:34.205658 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:34.205601 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" podStartSLOduration=2.205586202 podStartE2EDuration="2.205586202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:34.203938767 +0000 UTC m=+2480.464251293" watchObservedRunningTime="2026-07-30 21:02:34.205586202 +0000 UTC m=+2480.465898727" Jul 30 21:02:35.264428 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:35.264390 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:02:42.973895 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:42.973852 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:42.974390 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:42.973912 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:42.986426 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:42.986401 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:43.241262 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:43.241174 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:02:44.854636 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:44.854598 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:02:48.152838 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:48.152798 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:54.857170 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.857126 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75"] Jul 30 21:02:54.860645 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.860621 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:54.864216 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.864195 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12501c\"/\"kube-root-ca.crt\"" Jul 30 21:02:54.864307 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.864195 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12501c\"/\"default-dockercfg-q84js\"" Jul 30 21:02:54.864350 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.864197 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-12501c\"/\"test-model-12501c-kserve-self-signed-certs\"" Jul 30 21:02:54.864350 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.864256 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-12501c\"/\"openshift-service-ca.crt\"" Jul 30 21:02:54.869473 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.869450 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75"] Jul 30 21:02:54.940472 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.940420 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:02:54.953159 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.953125 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bgvn\" (UniqueName: \"kubernetes.io/projected/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-kube-api-access-2bgvn\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:54.953288 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.953167 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tls-certs\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:54.953288 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.953252 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tmp-dir\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:54.953379 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.953301 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-home\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:54.953379 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.953356 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-dshm\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:54.953453 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:54.953379 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-model-cache\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.054803 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.054742 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-home\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.054803 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.054805 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-dshm\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.055004 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.054913 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-model-cache\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.055004 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.054956 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bgvn\" (UniqueName: \"kubernetes.io/projected/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-kube-api-access-2bgvn\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.055004 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.054991 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tls-certs\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.055114 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.055068 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tmp-dir\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.055266 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.055241 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-home\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.055394 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.055323 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-model-cache\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.055394 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.055356 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tmp-dir\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.057057 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.057037 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-dshm\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.057587 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.057569 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tls-certs\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.063530 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.063511 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bgvn\" (UniqueName: \"kubernetes.io/projected/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-kube-api-access-2bgvn\") pod \"test-model-12501c-kserve-58c7d9f48f-5hm75\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.172130 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.172036 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:02:55.313495 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:55.313457 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75"] Jul 30 21:02:55.316461 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:02:55.316430 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecbf2262_0ddc_4edd_8132_fe12afbee0fa.slice/crio-38e235969861364aa8703d26122d7d31440e5722cd0fc7e51b95c12f1c2e548f WatchSource:0}: Error finding container 38e235969861364aa8703d26122d7d31440e5722cd0fc7e51b95c12f1c2e548f: Status 404 returned error can't find the container with id 38e235969861364aa8703d26122d7d31440e5722cd0fc7e51b95c12f1c2e548f Jul 30 21:02:56.290590 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:56.290557 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" event={"ID":"ecbf2262-0ddc-4edd-8132-fe12afbee0fa","Type":"ContainerStarted","Data":"5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a"} Jul 30 21:02:56.290974 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:56.290597 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" event={"ID":"ecbf2262-0ddc-4edd-8132-fe12afbee0fa","Type":"ContainerStarted","Data":"38e235969861364aa8703d26122d7d31440e5722cd0fc7e51b95c12f1c2e548f"} Jul 30 21:02:56.308905 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:56.308841 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" podStartSLOduration=2.308822579 podStartE2EDuration="2.308822579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:56.308637091 +0000 UTC m=+2502.568949618" watchObservedRunningTime="2026-07-30 21:02:56.308822579 +0000 UTC m=+2502.569135106" Jul 30 21:02:58.256616 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:02:58.256580 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:03:05.172503 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:05.172449 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:03:05.173171 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:05.172536 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:03:05.185422 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:05.185397 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:03:05.347911 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:05.347881 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:03:06.454221 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:06.454135 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:03:35.723974 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:35.723940 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5"] Jul 30 21:03:35.745927 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:35.745898 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:36.674212 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:36.674175 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75"] Jul 30 21:03:36.920462 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:36.920431 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:39.213239 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:39.213202 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:03:47.243167 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:47.243119 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:47.311219 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:47.311192 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:47.378871 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:47.378839 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:47.443013 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:47.442954 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:47.818446 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:47.818410 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:49.457676 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:49.457631 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:03:50.743252 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:50.743207 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" podUID="4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" containerName="main" containerID="cri-o://607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef" gracePeriod=45 Jul 30 21:03:50.984920 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:50.984895 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:03:51.062644 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062554 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tmp-dir\") pod \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " Jul 30 21:03:51.062644 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062599 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-dshm\") pod \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " Jul 30 21:03:51.062911 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062649 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-model-cache\") pod \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " Jul 30 21:03:51.062911 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062667 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-home\") pod \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " Jul 30 21:03:51.062911 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062729 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwllr\" (UniqueName: \"kubernetes.io/projected/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-kube-api-access-mwllr\") pod \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " Jul 30 21:03:51.062911 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062763 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tls-certs\") pod \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\" (UID: \"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b\") " Jul 30 21:03:51.062911 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062874 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" (UID: "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:51.063145 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062954 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" (UID: "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:51.063145 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.062996 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-home" (OuterVolumeSpecName: "home") pod "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" (UID: "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:51.063145 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.063057 2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tmp-dir\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:51.063145 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.063090 2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-model-cache\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:51.064945 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.064917 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-dshm" (OuterVolumeSpecName: "dshm") pod "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" (UID: "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:51.065032 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.064967 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" (UID: "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 21:03:51.065098 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.065072 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-kube-api-access-mwllr" (OuterVolumeSpecName: "kube-api-access-mwllr") pod "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" (UID: "4afb3434-4e4c-45e3-8ed8-a3b24d51b47b"). InnerVolumeSpecName "kube-api-access-mwllr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:03:51.164187 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.164143 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwllr\" (UniqueName: \"kubernetes.io/projected/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-kube-api-access-mwllr\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:51.164187 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.164180 2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-tls-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:51.164187 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.164192 2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-dshm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:51.164425 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.164201 2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b-home\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:51.551922 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.551887 2537 generic.go:356] "Generic (PLEG): container finished" podID="4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" containerID="607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef" exitCode=0 Jul 30 21:03:51.552101 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.551933 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" Jul 30 21:03:51.552101 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.551974 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" event={"ID":"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b","Type":"ContainerDied","Data":"607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef"} Jul 30 21:03:51.552101 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.552018 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5" event={"ID":"4afb3434-4e4c-45e3-8ed8-a3b24d51b47b","Type":"ContainerDied","Data":"826429cf1c4c5c22d30f86c1a1c546fc080d70880f49703c074009156f5f0ffd"} Jul 30 21:03:51.552101 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.552036 2537 scope.go:117] "RemoveContainer" containerID="607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef" Jul 30 21:03:51.561179 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.561159 2537 scope.go:117] "RemoveContainer" containerID="607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef" Jul 30 21:03:51.561489 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:03:51.561468 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef\": container with ID starting with 607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef not found: ID does not exist" containerID="607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef" Jul 30 21:03:51.561548 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.561497 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef"} err="failed to get container status \"607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef\": rpc error: code = NotFound desc = could not find container \"607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef\": container with ID starting with 607cddbf19475b4018e23e49f783d95e4558573b42eb4a21a90797faa0ac21ef not found: ID does not exist" Jul 30 21:03:51.573720 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.573689 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5"] Jul 30 21:03:51.579877 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.579846 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-72f733/test-model-72f733-kserve-5f8459fbf7-wfjp5"] Jul 30 21:03:51.693293 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.693248 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" podUID="ecbf2262-0ddc-4edd-8132-fe12afbee0fa" containerName="main" containerID="cri-o://5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a" gracePeriod=45 Jul 30 21:03:51.939755 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:51.939730 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:03:52.071522 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071429 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-model-cache\") pod \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " Jul 30 21:03:52.071522 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071485 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bgvn\" (UniqueName: \"kubernetes.io/projected/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-kube-api-access-2bgvn\") pod \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " Jul 30 21:03:52.071522 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071511 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-dshm\") pod \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " Jul 30 21:03:52.071761 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071634 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tmp-dir\") pod \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " Jul 30 21:03:52.071761 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071693 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-home\") pod \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " Jul 30 21:03:52.071761 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071729 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tls-certs\") pod \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\" (UID: \"ecbf2262-0ddc-4edd-8132-fe12afbee0fa\") " Jul 30 21:03:52.071761 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071739 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecbf2262-0ddc-4edd-8132-fe12afbee0fa" (UID: "ecbf2262-0ddc-4edd-8132-fe12afbee0fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:52.071955 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071934 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-home" (OuterVolumeSpecName: "home") pod "ecbf2262-0ddc-4edd-8132-fe12afbee0fa" (UID: "ecbf2262-0ddc-4edd-8132-fe12afbee0fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:52.071955 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.071942 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecbf2262-0ddc-4edd-8132-fe12afbee0fa" (UID: "ecbf2262-0ddc-4edd-8132-fe12afbee0fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:52.072122 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.072106 2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-model-cache\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:52.072187 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.072124 2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tmp-dir\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:52.072187 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.072138 2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-home\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:52.073863 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.073832 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-dshm" (OuterVolumeSpecName: "dshm") pod "ecbf2262-0ddc-4edd-8132-fe12afbee0fa" (UID: "ecbf2262-0ddc-4edd-8132-fe12afbee0fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:03:52.073978 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.073896 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecbf2262-0ddc-4edd-8132-fe12afbee0fa" (UID: "ecbf2262-0ddc-4edd-8132-fe12afbee0fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 21:03:52.073978 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.073897 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-kube-api-access-2bgvn" (OuterVolumeSpecName: "kube-api-access-2bgvn") pod "ecbf2262-0ddc-4edd-8132-fe12afbee0fa" (UID: "ecbf2262-0ddc-4edd-8132-fe12afbee0fa"). InnerVolumeSpecName "kube-api-access-2bgvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:03:52.136484 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.136447 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" path="/var/lib/kubelet/pods/4afb3434-4e4c-45e3-8ed8-a3b24d51b47b/volumes" Jul 30 21:03:52.173256 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.173211 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bgvn\" (UniqueName: \"kubernetes.io/projected/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-kube-api-access-2bgvn\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:52.173256 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.173250 2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-dshm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:52.173256 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.173260 2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecbf2262-0ddc-4edd-8132-fe12afbee0fa-tls-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:03:52.557388 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.557346 2537 generic.go:356] "Generic (PLEG): container finished" podID="ecbf2262-0ddc-4edd-8132-fe12afbee0fa" containerID="5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a" exitCode=0 Jul 30 21:03:52.557588 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.557413 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" Jul 30 21:03:52.557588 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.557432 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" event={"ID":"ecbf2262-0ddc-4edd-8132-fe12afbee0fa","Type":"ContainerDied","Data":"5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a"} Jul 30 21:03:52.557588 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.557474 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75" event={"ID":"ecbf2262-0ddc-4edd-8132-fe12afbee0fa","Type":"ContainerDied","Data":"38e235969861364aa8703d26122d7d31440e5722cd0fc7e51b95c12f1c2e548f"} Jul 30 21:03:52.557588 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.557496 2537 scope.go:117] "RemoveContainer" containerID="5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a" Jul 30 21:03:52.565793 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.565760 2537 scope.go:117] "RemoveContainer" containerID="5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a" Jul 30 21:03:52.566065 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:03:52.566039 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a\": container with ID starting with 5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a not found: ID does not exist" containerID="5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a" Jul 30 21:03:52.566115 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.566068 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a"} err="failed to get container status \"5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a\": rpc error: code = NotFound desc = could not find container \"5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a\": container with ID starting with 5ecf22aea34c29f5752b84d5a986c3c45a0f54e9c830b83f4e077d2ecded688a not found: ID does not exist" Jul 30 21:03:52.573875 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.573848 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75"] Jul 30 21:03:52.576000 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:52.575979 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-12501c/test-model-12501c-kserve-58c7d9f48f-5hm75"] Jul 30 21:03:53.697592 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:53.697550 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:53.828644 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:53.828613 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:53.962909 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:53.962760 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:03:54.129480 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:54.129441 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbf2262-0ddc-4edd-8132-fe12afbee0fa" path="/var/lib/kubelet/pods/ecbf2262-0ddc-4edd-8132-fe12afbee0fa/volumes" Jul 30 21:03:54.196048 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:03:54.196013 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:04.890429 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:04.890392 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:04.960056 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:04.960020 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:05.020557 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:05.020522 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:05.092053 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:05.092015 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:05.438177 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:05.438147 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:07.166648 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:07.166603 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:04:11.237198 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:11.237165 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:11.304311 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:11.304265 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:11.648721 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:11.648687 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:11.792278 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:11.792240 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:12.146438 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:12.146405 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:12.506377 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:12.506285 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:13.174656 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:13.174623 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:14.193701 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:14.193665 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:14.248500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:14.248455 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:15.519285 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:15.519243 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:15.616080 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:15.616017 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:16.257256 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:16.257217 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:16.937703 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:16.937664 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:17.034104 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:17.034062 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:18.754923 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:18.754886 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:04:40.237834 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:40.237735 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:40.815701 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:40.815637 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:41.052986 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:41.052944 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:41.195646 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:41.195608 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:44.194285 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:44.194250 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:44.293511 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:44.293473 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:45.364935 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:45.364897 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:45.793228 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:45.793136 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:46.127136 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:46.127087 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:46.227595 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:46.227561 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:04:47.848818 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:04:47.848767 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:05:09.264315 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:09.264275 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:26.612865 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:26.612831 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:27.858257 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:27.858222 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:27.925427 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:27.925395 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:28.025474 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:28.025434 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:28.121161 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:28.121077 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:28.447571 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:28.447493 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:30.303562 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:30.303526 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:05:34.135432 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:34.135401 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:34.239363 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:34.239326 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:34.428921 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:34.428836 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:34.675137 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:34.675103 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:50.282819 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:50.282764 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:50.367178 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:50.367145 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:50.467406 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:50.467376 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:50.532333 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:50.532296 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:50.915233 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:50.915207 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:52.163718 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:52.163687 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:05:56.643615 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:56.643582 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:56.802287 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:56.802234 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:56.944584 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:56.944504 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:57.194073 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:57.194042 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:57.565008 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:57.564976 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:57.917327 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:57.917289 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:05:58.066510 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:05:58.066477 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:00.716999 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:00.716960 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:00.810309 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:00.810280 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:01.839003 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:01.838967 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:02.025643 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:02.025602 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:02.338280 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:02.338244 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:02.496693 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:02.496649 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:04.357089 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:04.357054 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:06:25.711626 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:25.711590 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:26.088141 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:26.088055 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:26.421342 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:26.421305 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:26.523147 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:26.523112 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:29.810120 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:29.810083 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:29.917196 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:29.917165 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:30.930533 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:30.930499 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:31.285004 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:31.284910 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:31.420073 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:31.420033 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:31.522322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:31.522289 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:33.953961 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:33.953928 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:06:54.509984 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:54.509945 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:54.563043 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:54.563011 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:54.870401 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:54.870365 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:57.356921 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.356889 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk"] Jul 30 21:06:57.357389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.357268 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbf2262-0ddc-4edd-8132-fe12afbee0fa" containerName="main" Jul 30 21:06:57.357389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.357279 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbf2262-0ddc-4edd-8132-fe12afbee0fa" containerName="main" Jul 30 21:06:57.357389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.357298 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" containerName="main" Jul 30 21:06:57.357389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.357304 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" containerName="main" Jul 30 21:06:57.357389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.357370 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afb3434-4e4c-45e3-8ed8-a3b24d51b47b" containerName="main" Jul 30 21:06:57.357389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.357381 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbf2262-0ddc-4edd-8132-fe12afbee0fa" containerName="main" Jul 30 21:06:57.360324 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.360304 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.362839 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.362816 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-55df2b\"/\"openshift-service-ca.crt\"" Jul 30 21:06:57.363731 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.363707 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-55df2b\"/\"default-dockercfg-ht8pk\"" Jul 30 21:06:57.363857 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.363752 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-55df2b\"/\"kube-root-ca.crt\"" Jul 30 21:06:57.363857 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.363825 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-55df2b\"/\"auth-test-model-55df2b-kserve-self-signed-certs\"" Jul 30 21:06:57.369205 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.369179 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk"] Jul 30 21:06:57.387150 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.387118 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:06:57.411805 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.411748 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-model-cache\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.412007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.411823 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce285557-490e-4e2d-86d8-60b735379b36-tls-certs\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.412007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.411845 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-dshm\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.412007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.411886 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djnb\" (UniqueName: \"kubernetes.io/projected/ce285557-490e-4e2d-86d8-60b735379b36-kube-api-access-8djnb\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.412007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.411963 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-home\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.412007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.411999 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-tmp-dir\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.512609 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.512574 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce285557-490e-4e2d-86d8-60b735379b36-tls-certs\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.512822 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.512616 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-dshm\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.512822 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.512751 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djnb\" (UniqueName: \"kubernetes.io/projected/ce285557-490e-4e2d-86d8-60b735379b36-kube-api-access-8djnb\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.512953 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.512822 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-home\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.512953 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.512859 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-tmp-dir\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.512953 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.512922 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-model-cache\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.513231 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.513208 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-home\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.513457 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.513252 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-tmp-dir\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.513457 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.513272 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-model-cache\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.515043 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.515019 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-dshm\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.515379 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.515364 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce285557-490e-4e2d-86d8-60b735379b36-tls-certs\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.521554 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.521534 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djnb\" (UniqueName: \"kubernetes.io/projected/ce285557-490e-4e2d-86d8-60b735379b36-kube-api-access-8djnb\") pod \"auth-test-model-55df2b-kserve-5d8f85f887-sdctk\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.672705 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.672659 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:06:57.809129 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:57.809101 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk"] Jul 30 21:06:57.811633 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:06:57.811602 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce285557_490e_4e2d_86d8_60b735379b36.slice/crio-1d89ff0bf8885e61a77d655cdd366618b30110d61c2bd4d4e8e7b0efee5ce752 WatchSource:0}: Error finding container 1d89ff0bf8885e61a77d655cdd366618b30110d61c2bd4d4e8e7b0efee5ce752: Status 404 returned error can't find the container with id 1d89ff0bf8885e61a77d655cdd366618b30110d61c2bd4d4e8e7b0efee5ce752 Jul 30 21:06:58.416625 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:58.416585 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" event={"ID":"ce285557-490e-4e2d-86d8-60b735379b36","Type":"ContainerStarted","Data":"7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3"} Jul 30 21:06:58.417042 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:58.416636 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" event={"ID":"ce285557-490e-4e2d-86d8-60b735379b36","Type":"ContainerStarted","Data":"1d89ff0bf8885e61a77d655cdd366618b30110d61c2bd4d4e8e7b0efee5ce752"} Jul 30 21:06:58.432631 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:06:58.432579 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" podStartSLOduration=1.432564421 podStartE2EDuration="1.432564421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:06:58.43250663 +0000 UTC m=+2744.692819157" watchObservedRunningTime="2026-07-30 21:06:58.432564421 +0000 UTC m=+2744.692876946" Jul 30 21:07:00.056984 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:00.056942 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:07:07.673565 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:07.673524 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:07:07.673565 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:07.673576 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:07:07.685794 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:07.685756 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:07:08.472737 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:08.472703 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:07:22.988735 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:22.988702 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:07:25.662538 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:25.662498 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:07:50.261262 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:07:50.261219 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:08:05.960555 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:08:05.960520 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:08:16.669968 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:08:16.669931 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:08:32.256613 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:08:32.256578 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:08:45.755515 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:08:45.755481 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:08:55.862619 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:08:55.862580 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:09:12.154717 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:09:12.154636 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:09:25.854196 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:09:25.854157 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:09:35.370434 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:09:35.370396 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:09:42.459825 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:09:42.459786 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:09:50.408448 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:09:50.408416 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:09:57.461373 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:09:57.461297 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:10:00.657093 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:10:00.657050 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:10:00.754968 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:10:00.754926 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:10:08.946294 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:10:08.946260 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:10:23.758907 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:10:23.758871 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:10:37.164282 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:10:37.164202 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:10:48.465287 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:10:48.465248 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:11:03.663384 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:03.663351 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:11:17.273400 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:17.273362 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:11:21.216825 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:21.216795 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk"] Jul 30 21:11:21.238193 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:21.238159 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:11:22.164868 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:22.164827 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:11:22.525362 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:22.525239 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:11:27.851062 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:27.851019 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:11:29.552056 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.552018 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5"] Jul 30 21:11:29.555534 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.555511 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.557915 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.557887 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-55df2b\"/\"sub-test-model-55df2b-kserve-self-signed-certs\"" Jul 30 21:11:29.564245 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.564218 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5"] Jul 30 21:11:29.586972 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.586937 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-dshm\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.587146 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.587023 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-home\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.587146 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.587059 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-tmp-dir\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.587146 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.587091 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-model-cache\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.587275 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.587172 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bf13b-0809-4b1d-b1c0-b27b76c473af-tls-certs\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.587275 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.587230 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvpf\" (UniqueName: \"kubernetes.io/projected/393bf13b-0809-4b1d-b1c0-b27b76c473af-kube-api-access-llvpf\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.661902 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.661852 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:11:29.688379 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688342 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-home\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688379 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688382 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-tmp-dir\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688562 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688414 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-model-cache\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688562 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688470 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bf13b-0809-4b1d-b1c0-b27b76c473af-tls-certs\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688562 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688541 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvpf\" (UniqueName: \"kubernetes.io/projected/393bf13b-0809-4b1d-b1c0-b27b76c473af-kube-api-access-llvpf\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688689 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688565 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-dshm\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688858 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688834 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-tmp-dir\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688908 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688878 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-model-cache\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.688954 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.688917 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-home\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.690925 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.690897 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-dshm\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.691226 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.691206 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bf13b-0809-4b1d-b1c0-b27b76c473af-tls-certs\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.696603 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.696573 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvpf\" (UniqueName: \"kubernetes.io/projected/393bf13b-0809-4b1d-b1c0-b27b76c473af-kube-api-access-llvpf\") pod \"sub-test-model-55df2b-kserve-7667d49475-t55x5\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:29.865349 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:29.865305 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:30.003903 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:30.003872 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5"] Jul 30 21:11:30.006048 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:11:30.006017 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod393bf13b_0809_4b1d_b1c0_b27b76c473af.slice/crio-0c5414bf32d481e918b7132561798b9450109ed21100dea1a6284f982d616863 WatchSource:0}: Error finding container 0c5414bf32d481e918b7132561798b9450109ed21100dea1a6284f982d616863: Status 404 returned error can't find the container with id 0c5414bf32d481e918b7132561798b9450109ed21100dea1a6284f982d616863 Jul 30 21:11:30.665829 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:30.665790 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" event={"ID":"393bf13b-0809-4b1d-b1c0-b27b76c473af","Type":"ContainerStarted","Data":"4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028"} Jul 30 21:11:30.665829 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:30.665835 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" event={"ID":"393bf13b-0809-4b1d-b1c0-b27b76c473af","Type":"ContainerStarted","Data":"0c5414bf32d481e918b7132561798b9450109ed21100dea1a6284f982d616863"} Jul 30 21:11:30.683652 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:30.683590 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" podStartSLOduration=1.6835740879999999 podStartE2EDuration="1.683574088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:11:30.682014374 +0000 UTC m=+3016.942326900" watchObservedRunningTime="2026-07-30 21:11:30.683574088 +0000 UTC m=+3016.943886673" Jul 30 21:11:36.240199 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.240148 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" podUID="ce285557-490e-4e2d-86d8-60b735379b36" containerName="main" containerID="cri-o://7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3" gracePeriod=45 Jul 30 21:11:36.498570 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.498505 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:11:36.552546 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552514 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-home\") pod \"ce285557-490e-4e2d-86d8-60b735379b36\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " Jul 30 21:11:36.552740 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552575 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce285557-490e-4e2d-86d8-60b735379b36-tls-certs\") pod \"ce285557-490e-4e2d-86d8-60b735379b36\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " Jul 30 21:11:36.552740 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552628 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-dshm\") pod \"ce285557-490e-4e2d-86d8-60b735379b36\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " Jul 30 21:11:36.552740 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552651 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-tmp-dir\") pod \"ce285557-490e-4e2d-86d8-60b735379b36\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " Jul 30 21:11:36.552740 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552671 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-model-cache\") pod \"ce285557-490e-4e2d-86d8-60b735379b36\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " Jul 30 21:11:36.552740 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552714 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8djnb\" (UniqueName: \"kubernetes.io/projected/ce285557-490e-4e2d-86d8-60b735379b36-kube-api-access-8djnb\") pod \"ce285557-490e-4e2d-86d8-60b735379b36\" (UID: \"ce285557-490e-4e2d-86d8-60b735379b36\") " Jul 30 21:11:36.553021 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552882 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-home" (OuterVolumeSpecName: "home") pod "ce285557-490e-4e2d-86d8-60b735379b36" (UID: "ce285557-490e-4e2d-86d8-60b735379b36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:11:36.553021 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.552941 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce285557-490e-4e2d-86d8-60b735379b36" (UID: "ce285557-490e-4e2d-86d8-60b735379b36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:11:36.553133 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.553040 2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-home\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:11:36.553133 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.553060 2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-tmp-dir\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:11:36.553133 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.553099 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce285557-490e-4e2d-86d8-60b735379b36" (UID: "ce285557-490e-4e2d-86d8-60b735379b36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:11:36.555320 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.555298 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-dshm" (OuterVolumeSpecName: "dshm") pod "ce285557-490e-4e2d-86d8-60b735379b36" (UID: "ce285557-490e-4e2d-86d8-60b735379b36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:11:36.555448 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.555329 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce285557-490e-4e2d-86d8-60b735379b36-kube-api-access-8djnb" (OuterVolumeSpecName: "kube-api-access-8djnb") pod "ce285557-490e-4e2d-86d8-60b735379b36" (UID: "ce285557-490e-4e2d-86d8-60b735379b36"). InnerVolumeSpecName "kube-api-access-8djnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:11:36.555848 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.555826 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce285557-490e-4e2d-86d8-60b735379b36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce285557-490e-4e2d-86d8-60b735379b36" (UID: "ce285557-490e-4e2d-86d8-60b735379b36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 21:11:36.557604 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.557583 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:11:36.653578 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.653523 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djnb\" (UniqueName: \"kubernetes.io/projected/ce285557-490e-4e2d-86d8-60b735379b36-kube-api-access-8djnb\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:11:36.653578 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.653573 2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce285557-490e-4e2d-86d8-60b735379b36-tls-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:11:36.653578 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.653584 2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-dshm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:11:36.653578 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.653592 2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce285557-490e-4e2d-86d8-60b735379b36-model-cache\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:11:36.694829 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.694753 2537 generic.go:356] "Generic (PLEG): container finished" podID="ce285557-490e-4e2d-86d8-60b735379b36" containerID="7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3" exitCode=0 Jul 30 21:11:36.694829 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.694832 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" Jul 30 21:11:36.695081 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.694848 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" event={"ID":"ce285557-490e-4e2d-86d8-60b735379b36","Type":"ContainerDied","Data":"7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3"} Jul 30 21:11:36.695081 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.694918 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk" event={"ID":"ce285557-490e-4e2d-86d8-60b735379b36","Type":"ContainerDied","Data":"1d89ff0bf8885e61a77d655cdd366618b30110d61c2bd4d4e8e7b0efee5ce752"} Jul 30 21:11:36.695081 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.694941 2537 scope.go:117] "RemoveContainer" containerID="7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3" Jul 30 21:11:36.705656 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.705634 2537 scope.go:117] "RemoveContainer" containerID="7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3" Jul 30 21:11:36.705976 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:11:36.705954 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3\": container with ID starting with 7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3 not found: ID does not exist" containerID="7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3" Jul 30 21:11:36.706023 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.705985 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3"} err="failed to get container status \"7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3\": rpc error: code = NotFound desc = could not find container \"7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3\": container with ID starting with 7482c5d480469eb8f240b72a4cae6f6d43b258c4ab5c4c05d42051fc5ec9c5c3 not found: ID does not exist" Jul 30 21:11:36.717158 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.717132 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk"] Jul 30 21:11:36.722787 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:36.722755 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/auth-test-model-55df2b-kserve-5d8f85f887-sdctk"] Jul 30 21:11:38.129692 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:38.129655 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce285557-490e-4e2d-86d8-60b735379b36" path="/var/lib/kubelet/pods/ce285557-490e-4e2d-86d8-60b735379b36/volumes" Jul 30 21:11:39.866091 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:39.866059 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:39.866500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:39.866107 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:39.878492 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:39.878462 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:40.727244 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:40.727217 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:11:50.757083 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.757047 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9"] Jul 30 21:11:50.757515 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.757503 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce285557-490e-4e2d-86d8-60b735379b36" containerName="main" Jul 30 21:11:50.757566 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.757521 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce285557-490e-4e2d-86d8-60b735379b36" containerName="main" Jul 30 21:11:50.757607 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.757591 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce285557-490e-4e2d-86d8-60b735379b36" containerName="main" Jul 30 21:11:50.762102 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.762082 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.764669 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.764648 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"kube-root-ca.crt\"" Jul 30 21:11:50.765701 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.765677 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"sub-test-model-e226df-kserve-self-signed-certs\"" Jul 30 21:11:50.765802 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.765690 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"default-dockercfg-46q8q\"" Jul 30 21:11:50.765802 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.765720 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"openshift-service-ca.crt\"" Jul 30 21:11:50.769180 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.769049 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9"] Jul 30 21:11:50.851771 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.851719 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:11:50.879098 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.879066 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-model-cache\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.879297 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.879105 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-home\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.879297 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.879207 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-tmp-dir\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.879297 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.879277 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-dshm\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.879417 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.879308 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.879417 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.879335 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjl5q\" (UniqueName: \"kubernetes.io/projected/e775c551-8373-40dc-a011-e3c19cf569a8-kube-api-access-bjl5q\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980161 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980120 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-tmp-dir\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980366 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980196 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-dshm\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980366 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980223 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980366 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980256 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjl5q\" (UniqueName: \"kubernetes.io/projected/e775c551-8373-40dc-a011-e3c19cf569a8-kube-api-access-bjl5q\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980366 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980288 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-model-cache\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980366 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980319 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-home\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980638 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980610 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-tmp-dir\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980703 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980649 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-model-cache\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.980751 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.980697 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-home\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.982580 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.982555 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-dshm\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.982758 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.982741 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:50.987898 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:50.987878 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjl5q\" (UniqueName: \"kubernetes.io/projected/e775c551-8373-40dc-a011-e3c19cf569a8-kube-api-access-bjl5q\") pod \"sub-test-model-e226df-kserve-66754dbd97-tqqr9\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:51.071666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:51.071577 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:11:51.207064 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:51.207037 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9"] Jul 30 21:11:51.209959 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:11:51.209924 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode775c551_8373_40dc_a011_e3c19cf569a8.slice/crio-e29570565376b88a64f1f4349e80b2171b75abce86b4757cfb70c25809e3d87a WatchSource:0}: Error finding container e29570565376b88a64f1f4349e80b2171b75abce86b4757cfb70c25809e3d87a: Status 404 returned error can't find the container with id e29570565376b88a64f1f4349e80b2171b75abce86b4757cfb70c25809e3d87a Jul 30 21:11:51.765422 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:51.765387 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" event={"ID":"e775c551-8373-40dc-a011-e3c19cf569a8","Type":"ContainerStarted","Data":"13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d"} Jul 30 21:11:51.765422 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:51.765426 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" event={"ID":"e775c551-8373-40dc-a011-e3c19cf569a8","Type":"ContainerStarted","Data":"e29570565376b88a64f1f4349e80b2171b75abce86b4757cfb70c25809e3d87a"} Jul 30 21:11:51.784076 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:51.784028 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" podStartSLOduration=1.784011246 podStartE2EDuration="1.784011246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:11:51.781814081 +0000 UTC m=+3038.042126608" watchObservedRunningTime="2026-07-30 21:11:51.784011246 +0000 UTC m=+3038.044323771" Jul 30 21:11:53.653483 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:11:53.653436 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:12:01.072547 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:01.072503 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:12:01.073033 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:01.072679 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:12:01.090386 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:01.090342 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:12:01.825747 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:01.825715 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:12:12.499867 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:12.499762 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:12:12.566422 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:12.566379 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:12:29.451976 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:29.451941 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:12:40.055787 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:12:40.055737 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:13:08.452992 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:08.452953 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:13:13.181014 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:13.180981 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5"] Jul 30 21:13:13.212676 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:13.212644 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:13:14.167411 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:14.167373 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9"] Jul 30 21:13:14.186958 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:14.186931 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:13:14.249135 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:14.249097 2537 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-self-signed-certs: secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:14.249290 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:14.249170 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs podName:e775c551-8373-40dc-a011-e3c19cf569a8 nodeName:}" failed. No retries permitted until 2026-07-30 21:13:14.749155297 +0000 UTC m=+3121.009467801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs") pod "sub-test-model-e226df-kserve-66754dbd97-tqqr9" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8") : secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:14.753528 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:14.753493 2537 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-self-signed-certs: secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:14.753723 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:14.753562 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs podName:e775c551-8373-40dc-a011-e3c19cf569a8 nodeName:}" failed. No retries permitted until 2026-07-30 21:13:15.753547091 +0000 UTC m=+3122.013859594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs") pod "sub-test-model-e226df-kserve-66754dbd97-tqqr9" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8") : secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:15.763506 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:15.763471 2537 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-self-signed-certs: secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:15.763917 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:15.763569 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs podName:e775c551-8373-40dc-a011-e3c19cf569a8 nodeName:}" failed. No retries permitted until 2026-07-30 21:13:17.763550619 +0000 UTC m=+3124.023863136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs") pod "sub-test-model-e226df-kserve-66754dbd97-tqqr9" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8") : secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:17.778958 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:17.778920 2537 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-self-signed-certs: secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:17.779343 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:17.778997 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs podName:e775c551-8373-40dc-a011-e3c19cf569a8 nodeName:}" failed. No retries permitted until 2026-07-30 21:13:21.778980019 +0000 UTC m=+3128.039292532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs") pod "sub-test-model-e226df-kserve-66754dbd97-tqqr9" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8") : secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:18.852008 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:18.851970 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:13:21.813108 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:21.813071 2537 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-self-signed-certs: secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:21.813481 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:21.813150 2537 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs podName:e775c551-8373-40dc-a011-e3c19cf569a8 nodeName:}" failed. No retries permitted until 2026-07-30 21:13:29.813134584 +0000 UTC m=+3136.073447087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs") pod "sub-test-model-e226df-kserve-66754dbd97-tqqr9" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8") : secret "sub-test-model-e226df-kserve-self-signed-certs" not found Jul 30 21:13:23.896730 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:23.896684 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:13:26.601708 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:26.601672 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:13:28.200352 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.200304 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" podUID="393bf13b-0809-4b1d-b1c0-b27b76c473af" containerName="main" containerID="cri-o://4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028" gracePeriod=45 Jul 30 21:13:28.443234 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.443211 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:13:28.471762 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.471687 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-dshm\") pod \"393bf13b-0809-4b1d-b1c0-b27b76c473af\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " Jul 30 21:13:28.471762 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.471725 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-tmp-dir\") pod \"393bf13b-0809-4b1d-b1c0-b27b76c473af\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " Jul 30 21:13:28.471762 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.471763 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llvpf\" (UniqueName: \"kubernetes.io/projected/393bf13b-0809-4b1d-b1c0-b27b76c473af-kube-api-access-llvpf\") pod \"393bf13b-0809-4b1d-b1c0-b27b76c473af\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " Jul 30 21:13:28.472035 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.471828 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-home\") pod \"393bf13b-0809-4b1d-b1c0-b27b76c473af\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " Jul 30 21:13:28.472035 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.471857 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-model-cache\") pod \"393bf13b-0809-4b1d-b1c0-b27b76c473af\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " Jul 30 21:13:28.472035 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.471902 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bf13b-0809-4b1d-b1c0-b27b76c473af-tls-certs\") pod \"393bf13b-0809-4b1d-b1c0-b27b76c473af\" (UID: \"393bf13b-0809-4b1d-b1c0-b27b76c473af\") " Jul 30 21:13:28.472260 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.472051 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "393bf13b-0809-4b1d-b1c0-b27b76c473af" (UID: "393bf13b-0809-4b1d-b1c0-b27b76c473af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:28.472260 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.472162 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-model-cache" (OuterVolumeSpecName: "model-cache") pod "393bf13b-0809-4b1d-b1c0-b27b76c473af" (UID: "393bf13b-0809-4b1d-b1c0-b27b76c473af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:28.472260 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.472183 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-home" (OuterVolumeSpecName: "home") pod "393bf13b-0809-4b1d-b1c0-b27b76c473af" (UID: "393bf13b-0809-4b1d-b1c0-b27b76c473af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:28.472412 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.472315 2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-tmp-dir\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:28.472412 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.472335 2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-home\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:28.472412 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.472350 2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-model-cache\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:28.473960 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.473939 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-dshm" (OuterVolumeSpecName: "dshm") pod "393bf13b-0809-4b1d-b1c0-b27b76c473af" (UID: "393bf13b-0809-4b1d-b1c0-b27b76c473af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:28.474101 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.474082 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393bf13b-0809-4b1d-b1c0-b27b76c473af-kube-api-access-llvpf" (OuterVolumeSpecName: "kube-api-access-llvpf") pod "393bf13b-0809-4b1d-b1c0-b27b76c473af" (UID: "393bf13b-0809-4b1d-b1c0-b27b76c473af"). InnerVolumeSpecName "kube-api-access-llvpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:13:28.474349 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.474334 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393bf13b-0809-4b1d-b1c0-b27b76c473af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "393bf13b-0809-4b1d-b1c0-b27b76c473af" (UID: "393bf13b-0809-4b1d-b1c0-b27b76c473af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 21:13:28.573332 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.573292 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llvpf\" (UniqueName: \"kubernetes.io/projected/393bf13b-0809-4b1d-b1c0-b27b76c473af-kube-api-access-llvpf\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:28.573332 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.573323 2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393bf13b-0809-4b1d-b1c0-b27b76c473af-tls-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:28.573332 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:28.573332 2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393bf13b-0809-4b1d-b1c0-b27b76c473af-dshm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:29.186542 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.186504 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" podUID="e775c551-8373-40dc-a011-e3c19cf569a8" containerName="main" containerID="cri-o://13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d" gracePeriod=45 Jul 30 21:13:29.226129 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.226098 2537 generic.go:356] "Generic (PLEG): container finished" podID="393bf13b-0809-4b1d-b1c0-b27b76c473af" containerID="4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028" exitCode=0 Jul 30 21:13:29.226499 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.226144 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" Jul 30 21:13:29.226499 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.226183 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" event={"ID":"393bf13b-0809-4b1d-b1c0-b27b76c473af","Type":"ContainerDied","Data":"4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028"} Jul 30 21:13:29.226499 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.226225 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5" event={"ID":"393bf13b-0809-4b1d-b1c0-b27b76c473af","Type":"ContainerDied","Data":"0c5414bf32d481e918b7132561798b9450109ed21100dea1a6284f982d616863"} Jul 30 21:13:29.226499 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.226246 2537 scope.go:117] "RemoveContainer" containerID="4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028" Jul 30 21:13:29.283655 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.283625 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5"] Jul 30 21:13:29.287548 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.287526 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-55df2b/sub-test-model-55df2b-kserve-7667d49475-t55x5"] Jul 30 21:13:29.296805 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.296771 2537 scope.go:117] "RemoveContainer" containerID="4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028" Jul 30 21:13:29.297150 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:29.297130 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028\": container with ID starting with 4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028 not found: ID does not exist" containerID="4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028" Jul 30 21:13:29.297204 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.297159 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028"} err="failed to get container status \"4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028\": rpc error: code = NotFound desc = could not find container \"4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028\": container with ID starting with 4663e5f81bb7585c2b95f5535fe68179bc2b8e607045481c8f73ecfc0057c028 not found: ID does not exist" Jul 30 21:13:29.437801 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.437719 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:13:29.482463 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482431 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-model-cache\") pod \"e775c551-8373-40dc-a011-e3c19cf569a8\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " Jul 30 21:13:29.482616 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482469 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjl5q\" (UniqueName: \"kubernetes.io/projected/e775c551-8373-40dc-a011-e3c19cf569a8-kube-api-access-bjl5q\") pod \"e775c551-8373-40dc-a011-e3c19cf569a8\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " Jul 30 21:13:29.482616 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482522 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-tmp-dir\") pod \"e775c551-8373-40dc-a011-e3c19cf569a8\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " Jul 30 21:13:29.482616 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482594 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs\") pod \"e775c551-8373-40dc-a011-e3c19cf569a8\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " Jul 30 21:13:29.482812 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482638 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-dshm\") pod \"e775c551-8373-40dc-a011-e3c19cf569a8\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " Jul 30 21:13:29.482812 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482664 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-home\") pod \"e775c551-8373-40dc-a011-e3c19cf569a8\" (UID: \"e775c551-8373-40dc-a011-e3c19cf569a8\") " Jul 30 21:13:29.482812 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482711 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "e775c551-8373-40dc-a011-e3c19cf569a8" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:29.482956 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.482916 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-home" (OuterVolumeSpecName: "home") pod "e775c551-8373-40dc-a011-e3c19cf569a8" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:29.483175 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.483132 2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-model-cache\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:29.483175 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.483146 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e775c551-8373-40dc-a011-e3c19cf569a8" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:29.483175 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.483155 2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-home\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:29.484922 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.484901 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e775c551-8373-40dc-a011-e3c19cf569a8-kube-api-access-bjl5q" (OuterVolumeSpecName: "kube-api-access-bjl5q") pod "e775c551-8373-40dc-a011-e3c19cf569a8" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8"). InnerVolumeSpecName "kube-api-access-bjl5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:13:29.485435 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.485403 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e775c551-8373-40dc-a011-e3c19cf569a8" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 21:13:29.485518 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.485430 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-dshm" (OuterVolumeSpecName: "dshm") pod "e775c551-8373-40dc-a011-e3c19cf569a8" (UID: "e775c551-8373-40dc-a011-e3c19cf569a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:13:29.584082 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.584047 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjl5q\" (UniqueName: \"kubernetes.io/projected/e775c551-8373-40dc-a011-e3c19cf569a8-kube-api-access-bjl5q\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:29.584082 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.584075 2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-tmp-dir\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:29.584271 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.584092 2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e775c551-8373-40dc-a011-e3c19cf569a8-tls-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:29.584271 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:29.584102 2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e775c551-8373-40dc-a011-e3c19cf569a8-dshm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:13:30.130072 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.130037 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393bf13b-0809-4b1d-b1c0-b27b76c473af" path="/var/lib/kubelet/pods/393bf13b-0809-4b1d-b1c0-b27b76c473af/volumes" Jul 30 21:13:30.237169 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.237136 2537 generic.go:356] "Generic (PLEG): container finished" podID="e775c551-8373-40dc-a011-e3c19cf569a8" containerID="13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d" exitCode=0 Jul 30 21:13:30.237633 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.237193 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" Jul 30 21:13:30.237633 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.237221 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" event={"ID":"e775c551-8373-40dc-a011-e3c19cf569a8","Type":"ContainerDied","Data":"13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d"} Jul 30 21:13:30.237633 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.237271 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9" event={"ID":"e775c551-8373-40dc-a011-e3c19cf569a8","Type":"ContainerDied","Data":"e29570565376b88a64f1f4349e80b2171b75abce86b4757cfb70c25809e3d87a"} Jul 30 21:13:30.237633 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.237294 2537 scope.go:117] "RemoveContainer" containerID="13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d" Jul 30 21:13:30.247706 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.247683 2537 scope.go:117] "RemoveContainer" containerID="13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d" Jul 30 21:13:30.248198 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:13:30.248180 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d\": container with ID starting with 13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d not found: ID does not exist" containerID="13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d" Jul 30 21:13:30.248293 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.248205 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d"} err="failed to get container status \"13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d\": rpc error: code = NotFound desc = could not find container \"13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d\": container with ID starting with 13acc1d7256c57f57fe80b92737c20dea5f8fbc6a03af05c7b9b6813e047261d not found: ID does not exist" Jul 30 21:13:30.270651 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.270625 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9"] Jul 30 21:13:30.275854 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:30.275827 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/sub-test-model-e226df-kserve-66754dbd97-tqqr9"] Jul 30 21:13:32.129569 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:32.129535 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e775c551-8373-40dc-a011-e3c19cf569a8" path="/var/lib/kubelet/pods/e775c551-8373-40dc-a011-e3c19cf569a8/volumes" Jul 30 21:13:53.859715 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.859679 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf"] Jul 30 21:13:53.860209 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.860072 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393bf13b-0809-4b1d-b1c0-b27b76c473af" containerName="main" Jul 30 21:13:53.860209 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.860085 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="393bf13b-0809-4b1d-b1c0-b27b76c473af" containerName="main" Jul 30 21:13:53.860209 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.860105 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e775c551-8373-40dc-a011-e3c19cf569a8" containerName="main" Jul 30 21:13:53.860209 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.860110 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="e775c551-8373-40dc-a011-e3c19cf569a8" containerName="main" Jul 30 21:13:53.860209 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.860162 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="393bf13b-0809-4b1d-b1c0-b27b76c473af" containerName="main" Jul 30 21:13:53.860209 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.860170 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="e775c551-8373-40dc-a011-e3c19cf569a8" containerName="main" Jul 30 21:13:53.864529 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.864512 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:53.867266 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.867242 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"openshift-service-ca.crt\"" Jul 30 21:13:53.867373 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.867249 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"kube-root-ca.crt\"" Jul 30 21:13:53.868417 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.868398 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"rate-test-model-e226df-kserve-self-signed-certs\"" Jul 30 21:13:53.868484 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.868422 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e226df\"/\"default-dockercfg-46q8q\"" Jul 30 21:13:53.873654 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.873631 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf"] Jul 30 21:13:53.883128 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.883097 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:13:53.992608 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.992576 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knch5\" (UniqueName: \"kubernetes.io/projected/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-kube-api-access-knch5\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:53.992836 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.992624 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-model-cache\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:53.992836 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.992691 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-home\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:53.992836 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.992751 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-dshm\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:53.992836 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.992830 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tmp-dir\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:53.993032 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:53.992867 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tls-certs\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.093574 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.093538 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knch5\" (UniqueName: \"kubernetes.io/projected/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-kube-api-access-knch5\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.093574 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.093579 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-model-cache\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.093873 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.093604 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-home\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.093873 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.093622 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-dshm\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.093873 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.093653 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tmp-dir\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.093873 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.093672 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tls-certs\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.094094 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.094041 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-home\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.094094 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.094078 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-model-cache\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.094173 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.094117 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tmp-dir\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.095959 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.095941 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-dshm\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.096131 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.096113 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tls-certs\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.102240 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.102218 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knch5\" (UniqueName: \"kubernetes.io/projected/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-kube-api-access-knch5\") pod \"rate-test-model-e226df-kserve-6b7949976c-9btqf\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.174717 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.174632 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:13:54.304241 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.304213 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf"] Jul 30 21:13:54.306128 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:13:54.306100 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e7fda69_87c2_4bdd_a359_83262a4eb9a4.slice/crio-edeec6fd41c71544bff748053a8e99b8d9bd84deccf495b8ec175ca82292fa5f WatchSource:0}: Error finding container edeec6fd41c71544bff748053a8e99b8d9bd84deccf495b8ec175ca82292fa5f: Status 404 returned error can't find the container with id edeec6fd41c71544bff748053a8e99b8d9bd84deccf495b8ec175ca82292fa5f Jul 30 21:13:54.353227 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:54.353200 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" event={"ID":"7e7fda69-87c2-4bdd-a359-83262a4eb9a4","Type":"ContainerStarted","Data":"edeec6fd41c71544bff748053a8e99b8d9bd84deccf495b8ec175ca82292fa5f"} Jul 30 21:13:55.359409 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:55.359370 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" event={"ID":"7e7fda69-87c2-4bdd-a359-83262a4eb9a4","Type":"ContainerStarted","Data":"1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d"} Jul 30 21:13:55.378421 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:55.378376 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" podStartSLOduration=2.378335666 podStartE2EDuration="2.378335666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:13:55.377255073 +0000 UTC m=+3161.637567622" watchObservedRunningTime="2026-07-30 21:13:55.378335666 +0000 UTC m=+3161.638648192" Jul 30 21:13:56.471517 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:13:56.471482 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:14:04.175570 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:14:04.175532 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:14:04.175981 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:14:04.175584 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:14:04.188033 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:14:04.188003 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:14:04.411111 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:14:04.411082 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:14:42.159722 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:14:42.155568 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:14:50.161824 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:14:50.161788 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:15:21.761103 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:21.761073 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:15:27.572217 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:27.572176 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:28.387680 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:28.387644 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf"] Jul 30 21:15:29.160207 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.160152 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:29.394752 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.394724 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7281-140466090881456"] Jul 30 21:15:29.397961 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.397942 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090881456" Jul 30 21:15:29.400282 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.400262 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p4qfb\"" Jul 30 21:15:29.404136 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.404116 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090881456"] Jul 30 21:15:29.530765 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.530685 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsck6\" (UniqueName: \"kubernetes.io/projected/1cd9ad9e-5424-46d6-9afc-99a60f68afe7-kube-api-access-tsck6\") pod \"test-curl-7281-140466090881456\" (UID: \"1cd9ad9e-5424-46d6-9afc-99a60f68afe7\") " pod="opendatahub/test-curl-7281-140466090881456" Jul 30 21:15:29.631719 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.631688 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsck6\" (UniqueName: \"kubernetes.io/projected/1cd9ad9e-5424-46d6-9afc-99a60f68afe7-kube-api-access-tsck6\") pod \"test-curl-7281-140466090881456\" (UID: \"1cd9ad9e-5424-46d6-9afc-99a60f68afe7\") " pod="opendatahub/test-curl-7281-140466090881456" Jul 30 21:15:29.639740 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.639715 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsck6\" (UniqueName: \"kubernetes.io/projected/1cd9ad9e-5424-46d6-9afc-99a60f68afe7-kube-api-access-tsck6\") pod \"test-curl-7281-140466090881456\" (UID: \"1cd9ad9e-5424-46d6-9afc-99a60f68afe7\") " pod="opendatahub/test-curl-7281-140466090881456" Jul 30 21:15:29.707532 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.707489 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090881456" Jul 30 21:15:29.837612 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.837583 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090881456"] Jul 30 21:15:29.840238 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:15:29.840208 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cd9ad9e_5424_46d6_9afc_99a60f68afe7.slice/crio-a4c7dd38479013939b2bfd7c99d27fff2809c9fd93958332ba9f4f901a68afd9 WatchSource:0}: Error finding container a4c7dd38479013939b2bfd7c99d27fff2809c9fd93958332ba9f4f901a68afd9: Status 404 returned error can't find the container with id a4c7dd38479013939b2bfd7c99d27fff2809c9fd93958332ba9f4f901a68afd9 Jul 30 21:15:29.841593 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:29.841576 2537 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 21:15:30.761490 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:30.761454 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:15:30.800242 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:30.800151 2537 generic.go:356] "Generic (PLEG): container finished" podID="1cd9ad9e-5424-46d6-9afc-99a60f68afe7" containerID="a592d7d4a2a5ef9659b4e1d9ead8b8d77785e2263020a6886e019e104071ebc9" exitCode=0 Jul 30 21:15:30.800399 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:30.800302 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090881456" event={"ID":"1cd9ad9e-5424-46d6-9afc-99a60f68afe7","Type":"ContainerDied","Data":"a592d7d4a2a5ef9659b4e1d9ead8b8d77785e2263020a6886e019e104071ebc9"} Jul 30 21:15:30.800399 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:30.800366 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090881456" event={"ID":"1cd9ad9e-5424-46d6-9afc-99a60f68afe7","Type":"ContainerStarted","Data":"a4c7dd38479013939b2bfd7c99d27fff2809c9fd93958332ba9f4f901a68afd9"} Jul 30 21:15:31.939815 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:31.939792 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090881456" Jul 30 21:15:31.960846 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:31.960827 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7281-140466090881456_1cd9ad9e-5424-46d6-9afc-99a60f68afe7/test-curl-7281-140466090881456/0.log" Jul 30 21:15:31.996135 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:31.996108 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7281-140466090881456"] Jul 30 21:15:32.010336 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.009913 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7281-140466090881456"] Jul 30 21:15:32.055099 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.055074 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsck6\" (UniqueName: \"kubernetes.io/projected/1cd9ad9e-5424-46d6-9afc-99a60f68afe7-kube-api-access-tsck6\") pod \"1cd9ad9e-5424-46d6-9afc-99a60f68afe7\" (UID: \"1cd9ad9e-5424-46d6-9afc-99a60f68afe7\") " Jul 30 21:15:32.057157 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.057100 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd9ad9e-5424-46d6-9afc-99a60f68afe7-kube-api-access-tsck6" (OuterVolumeSpecName: "kube-api-access-tsck6") pod "1cd9ad9e-5424-46d6-9afc-99a60f68afe7" (UID: "1cd9ad9e-5424-46d6-9afc-99a60f68afe7"). InnerVolumeSpecName "kube-api-access-tsck6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:15:32.130067 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.130034 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd9ad9e-5424-46d6-9afc-99a60f68afe7" path="/var/lib/kubelet/pods/1cd9ad9e-5424-46d6-9afc-99a60f68afe7/volumes" Jul 30 21:15:32.156244 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.156217 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsck6\" (UniqueName: \"kubernetes.io/projected/1cd9ad9e-5424-46d6-9afc-99a60f68afe7-kube-api-access-tsck6\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:32.176586 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.176553 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7281-140466090884272"] Jul 30 21:15:32.176937 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.176921 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd9ad9e-5424-46d6-9afc-99a60f68afe7" containerName="test-curl-7281-140466090881456" Jul 30 21:15:32.177016 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.176956 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd9ad9e-5424-46d6-9afc-99a60f68afe7" containerName="test-curl-7281-140466090881456" Jul 30 21:15:32.177086 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.177073 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd9ad9e-5424-46d6-9afc-99a60f68afe7" containerName="test-curl-7281-140466090881456" Jul 30 21:15:32.180068 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.180051 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:32.197601 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.197581 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090884272"] Jul 30 21:15:32.358773 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.358743 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psk54\" (UniqueName: \"kubernetes.io/projected/9a674262-0372-4af6-b627-68d30ace2b1c-kube-api-access-psk54\") pod \"test-curl-7281-140466090884272\" (UID: \"9a674262-0372-4af6-b627-68d30ace2b1c\") " pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:32.460244 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.460207 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psk54\" (UniqueName: \"kubernetes.io/projected/9a674262-0372-4af6-b627-68d30ace2b1c-kube-api-access-psk54\") pod \"test-curl-7281-140466090884272\" (UID: \"9a674262-0372-4af6-b627-68d30ace2b1c\") " pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:32.469083 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.469053 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psk54\" (UniqueName: \"kubernetes.io/projected/9a674262-0372-4af6-b627-68d30ace2b1c-kube-api-access-psk54\") pod \"test-curl-7281-140466090884272\" (UID: \"9a674262-0372-4af6-b627-68d30ace2b1c\") " pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:32.488213 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.488191 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:32.622456 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.622345 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090884272"] Jul 30 21:15:32.624692 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:15:32.624663 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a674262_0372_4af6_b627_68d30ace2b1c.slice/crio-4b987a68806ce22fe646d56b4f4996f8cb4437262c9b1af98d2560094214c3db WatchSource:0}: Error finding container 4b987a68806ce22fe646d56b4f4996f8cb4437262c9b1af98d2560094214c3db: Status 404 returned error can't find the container with id 4b987a68806ce22fe646d56b4f4996f8cb4437262c9b1af98d2560094214c3db Jul 30 21:15:32.811732 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.811699 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090884272" event={"ID":"9a674262-0372-4af6-b627-68d30ace2b1c","Type":"ContainerStarted","Data":"4b987a68806ce22fe646d56b4f4996f8cb4437262c9b1af98d2560094214c3db"} Jul 30 21:15:32.812856 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.812836 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090881456" Jul 30 21:15:32.812973 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:32.812837 2537 scope.go:117] "RemoveContainer" containerID="a592d7d4a2a5ef9659b4e1d9ead8b8d77785e2263020a6886e019e104071ebc9" Jul 30 21:15:33.818769 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:33.818737 2537 generic.go:356] "Generic (PLEG): container finished" podID="9a674262-0372-4af6-b627-68d30ace2b1c" containerID="eb8fe33fd90e5a417565b0068aa1e4a3a6b34e9e8ee17ae1bb858bf6ec83291e" exitCode=0 Jul 30 21:15:33.819180 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:33.818815 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090884272" event={"ID":"9a674262-0372-4af6-b627-68d30ace2b1c","Type":"ContainerDied","Data":"eb8fe33fd90e5a417565b0068aa1e4a3a6b34e9e8ee17ae1bb858bf6ec83291e"} Jul 30 21:15:34.962433 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:34.962409 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:34.983981 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:34.983961 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7281-140466090884272_9a674262-0372-4af6-b627-68d30ace2b1c/test-curl-7281-140466090884272/0.log" Jul 30 21:15:35.008749 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.008724 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7281-140466090884272"] Jul 30 21:15:35.012398 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.012378 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7281-140466090884272"] Jul 30 21:15:35.081693 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.081607 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psk54\" (UniqueName: \"kubernetes.io/projected/9a674262-0372-4af6-b627-68d30ace2b1c-kube-api-access-psk54\") pod \"9a674262-0372-4af6-b627-68d30ace2b1c\" (UID: \"9a674262-0372-4af6-b627-68d30ace2b1c\") " Jul 30 21:15:35.083605 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.083575 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a674262-0372-4af6-b627-68d30ace2b1c-kube-api-access-psk54" (OuterVolumeSpecName: "kube-api-access-psk54") pod "9a674262-0372-4af6-b627-68d30ace2b1c" (UID: "9a674262-0372-4af6-b627-68d30ace2b1c"). InnerVolumeSpecName "kube-api-access-psk54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:15:35.178658 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.178627 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:35.179046 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.179032 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a674262-0372-4af6-b627-68d30ace2b1c" containerName="test-curl-7281-140466090884272" Jul 30 21:15:35.179087 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.179048 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a674262-0372-4af6-b627-68d30ace2b1c" containerName="test-curl-7281-140466090884272" Jul 30 21:15:35.179125 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.179118 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a674262-0372-4af6-b627-68d30ace2b1c" containerName="test-curl-7281-140466090884272" Jul 30 21:15:35.182070 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.182047 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:35.182423 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.182403 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psk54\" (UniqueName: \"kubernetes.io/projected/9a674262-0372-4af6-b627-68d30ace2b1c-kube-api-access-psk54\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:35.186395 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.186370 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:35.283166 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.283134 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghqf\" (UniqueName: \"kubernetes.io/projected/9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d-kube-api-access-jghqf\") pod \"test-curl-7281-140466090978448\" (UID: \"9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d\") " pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:35.384461 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.384430 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghqf\" (UniqueName: \"kubernetes.io/projected/9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d-kube-api-access-jghqf\") pod \"test-curl-7281-140466090978448\" (UID: \"9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d\") " pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:35.392468 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.392448 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghqf\" (UniqueName: \"kubernetes.io/projected/9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d-kube-api-access-jghqf\") pod \"test-curl-7281-140466090978448\" (UID: \"9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d\") " pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:35.492041 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.492009 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:35.617939 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.617896 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:35.620109 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:15:35.620078 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a93aa1d_c394_4463_a9d7_6ff9e4e6fa8d.slice/crio-02271904c9db8804dde5c76865593b6940964c29dc2cdd9436c6c159a508c94f WatchSource:0}: Error finding container 02271904c9db8804dde5c76865593b6940964c29dc2cdd9436c6c159a508c94f: Status 404 returned error can't find the container with id 02271904c9db8804dde5c76865593b6940964c29dc2cdd9436c6c159a508c94f Jul 30 21:15:35.831007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.830970 2537 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b987a68806ce22fe646d56b4f4996f8cb4437262c9b1af98d2560094214c3db" Jul 30 21:15:35.831007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.830973 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:35.832162 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:35.832137 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090978448" event={"ID":"9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d","Type":"ContainerStarted","Data":"02271904c9db8804dde5c76865593b6940964c29dc2cdd9436c6c159a508c94f"} Jul 30 21:15:35.841666 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:15:35.841642 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7281-140466090884272\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="9a674262-0372-4af6-b627-68d30ace2b1c" pod="opendatahub/test-curl-7281-140466090884272" Jul 30 21:15:36.129841 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:36.129807 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a674262-0372-4af6-b627-68d30ace2b1c" path="/var/lib/kubelet/pods/9a674262-0372-4af6-b627-68d30ace2b1c/volumes" Jul 30 21:15:36.843469 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:36.843430 2537 generic.go:356] "Generic (PLEG): container finished" podID="9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d" containerID="eaf7d4d0bc57e2c4d8efaaa9846f7288cfb90b111eb3e801d67a00253cd9ac72" exitCode=0 Jul 30 21:15:36.843639 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:36.843510 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090978448" event={"ID":"9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d","Type":"ContainerDied","Data":"eaf7d4d0bc57e2c4d8efaaa9846f7288cfb90b111eb3e801d67a00253cd9ac72"} Jul 30 21:15:37.977861 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:37.977838 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:37.997871 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:37.997847 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7281-140466090978448_9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d/test-curl-7281-140466090978448/0.log" Jul 30 21:15:38.051649 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.051618 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:38.056857 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.056838 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:38.108717 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.108697 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jghqf\" (UniqueName: \"kubernetes.io/projected/9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d-kube-api-access-jghqf\") pod \"9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d\" (UID: \"9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d\") " Jul 30 21:15:38.110858 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.110836 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d-kube-api-access-jghqf" (OuterVolumeSpecName: "kube-api-access-jghqf") pod "9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d" (UID: "9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d"). InnerVolumeSpecName "kube-api-access-jghqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:15:38.130341 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.130316 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d" path="/var/lib/kubelet/pods/9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d/volumes" Jul 30 21:15:38.177740 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.177709 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:38.178063 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.178050 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d" containerName="test-curl-7281-140466090978448" Jul 30 21:15:38.178108 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.178065 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d" containerName="test-curl-7281-140466090978448" Jul 30 21:15:38.178144 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.178130 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d" containerName="test-curl-7281-140466090978448" Jul 30 21:15:38.180961 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.180946 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:38.185412 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.185359 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:38.209759 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.209732 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jghqf\" (UniqueName: \"kubernetes.io/projected/9a93aa1d-c394-4463-a9d7-6ff9e4e6fa8d-kube-api-access-jghqf\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:38.310382 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.310347 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rgc\" (UniqueName: \"kubernetes.io/projected/1c9fab32-f50d-4b18-a4ab-ab3066853008-kube-api-access-h4rgc\") pod \"test-curl-7281-140466090978448\" (UID: \"1c9fab32-f50d-4b18-a4ab-ab3066853008\") " pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:38.411699 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.411625 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rgc\" (UniqueName: \"kubernetes.io/projected/1c9fab32-f50d-4b18-a4ab-ab3066853008-kube-api-access-h4rgc\") pod \"test-curl-7281-140466090978448\" (UID: \"1c9fab32-f50d-4b18-a4ab-ab3066853008\") " pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:38.420342 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.420321 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rgc\" (UniqueName: \"kubernetes.io/projected/1c9fab32-f50d-4b18-a4ab-ab3066853008-kube-api-access-h4rgc\") pod \"test-curl-7281-140466090978448\" (UID: \"1c9fab32-f50d-4b18-a4ab-ab3066853008\") " pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:38.492127 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.492095 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:38.622694 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.622665 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:38.624655 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:15:38.624617 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c9fab32_f50d_4b18_a4ab_ab3066853008.slice/crio-aedd3791814746c9c0784ec6ea72aa3c6dd673f9810d5e43a06b00e3979493bc WatchSource:0}: Error finding container aedd3791814746c9c0784ec6ea72aa3c6dd673f9810d5e43a06b00e3979493bc: Status 404 returned error can't find the container with id aedd3791814746c9c0784ec6ea72aa3c6dd673f9810d5e43a06b00e3979493bc Jul 30 21:15:38.854468 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.854377 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090978448" event={"ID":"1c9fab32-f50d-4b18-a4ab-ab3066853008","Type":"ContainerStarted","Data":"aedd3791814746c9c0784ec6ea72aa3c6dd673f9810d5e43a06b00e3979493bc"} Jul 30 21:15:38.855473 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.855446 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:38.855577 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:38.855449 2537 scope.go:117] "RemoveContainer" containerID="eaf7d4d0bc57e2c4d8efaaa9846f7288cfb90b111eb3e801d67a00253cd9ac72" Jul 30 21:15:39.861989 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:39.861952 2537 generic.go:356] "Generic (PLEG): container finished" podID="1c9fab32-f50d-4b18-a4ab-ab3066853008" containerID="9dbff54fed46043c9d268000d08d985198adc28068e7332ceb3964c65a8d53c9" exitCode=0 Jul 30 21:15:39.862396 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:39.862039 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466090978448" event={"ID":"1c9fab32-f50d-4b18-a4ab-ab3066853008","Type":"ContainerDied","Data":"9dbff54fed46043c9d268000d08d985198adc28068e7332ceb3964c65a8d53c9"} Jul 30 21:15:41.003401 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.003373 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:41.023660 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.023628 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7281-140466090978448_1c9fab32-f50d-4b18-a4ab-ab3066853008/test-curl-7281-140466090978448/0.log" Jul 30 21:15:41.051643 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.051620 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:41.054833 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.054807 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7281-140466090978448"] Jul 30 21:15:41.134613 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.134541 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4rgc\" (UniqueName: \"kubernetes.io/projected/1c9fab32-f50d-4b18-a4ab-ab3066853008-kube-api-access-h4rgc\") pod \"1c9fab32-f50d-4b18-a4ab-ab3066853008\" (UID: \"1c9fab32-f50d-4b18-a4ab-ab3066853008\") " Jul 30 21:15:41.136644 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.136622 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9fab32-f50d-4b18-a4ab-ab3066853008-kube-api-access-h4rgc" (OuterVolumeSpecName: "kube-api-access-h4rgc") pod "1c9fab32-f50d-4b18-a4ab-ab3066853008" (UID: "1c9fab32-f50d-4b18-a4ab-ab3066853008"). InnerVolumeSpecName "kube-api-access-h4rgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:15:41.182095 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.182068 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:41.182427 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.182415 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9fab32-f50d-4b18-a4ab-ab3066853008" containerName="test-curl-7281-140466090978448" Jul 30 21:15:41.182474 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.182429 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9fab32-f50d-4b18-a4ab-ab3066853008" containerName="test-curl-7281-140466090978448" Jul 30 21:15:41.182508 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.182483 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9fab32-f50d-4b18-a4ab-ab3066853008" containerName="test-curl-7281-140466090978448" Jul 30 21:15:41.185322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.185307 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:41.192746 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.192719 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:41.235187 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.235154 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rgc\" (UniqueName: \"kubernetes.io/projected/1c9fab32-f50d-4b18-a4ab-ab3066853008-kube-api-access-h4rgc\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:41.335997 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.335958 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6xg\" (UniqueName: \"kubernetes.io/projected/e79e2260-07f8-4075-949f-f377dab31998-kube-api-access-wq6xg\") pod \"test-curl-7281-140466091077680\" (UID: \"e79e2260-07f8-4075-949f-f377dab31998\") " pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:41.437183 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.437099 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6xg\" (UniqueName: \"kubernetes.io/projected/e79e2260-07f8-4075-949f-f377dab31998-kube-api-access-wq6xg\") pod \"test-curl-7281-140466091077680\" (UID: \"e79e2260-07f8-4075-949f-f377dab31998\") " pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:41.445958 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.445931 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6xg\" (UniqueName: \"kubernetes.io/projected/e79e2260-07f8-4075-949f-f377dab31998-kube-api-access-wq6xg\") pod \"test-curl-7281-140466091077680\" (UID: \"e79e2260-07f8-4075-949f-f377dab31998\") " pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:41.495944 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.495909 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:41.624641 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.624614 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:41.626611 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:15:41.626585 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode79e2260_07f8_4075_949f_f377dab31998.slice/crio-cc22c6f805b452475f086ee1b492281872d8bff0ae40ddd63e94c9f0892ce438 WatchSource:0}: Error finding container cc22c6f805b452475f086ee1b492281872d8bff0ae40ddd63e94c9f0892ce438: Status 404 returned error can't find the container with id cc22c6f805b452475f086ee1b492281872d8bff0ae40ddd63e94c9f0892ce438 Jul 30 21:15:41.872897 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.872872 2537 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aedd3791814746c9c0784ec6ea72aa3c6dd673f9810d5e43a06b00e3979493bc" Jul 30 21:15:41.872897 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.872880 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:41.874107 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:41.874083 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466091077680" event={"ID":"e79e2260-07f8-4075-949f-f377dab31998","Type":"ContainerStarted","Data":"cc22c6f805b452475f086ee1b492281872d8bff0ae40ddd63e94c9f0892ce438"} Jul 30 21:15:41.883605 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:15:41.883578 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7281-140466090978448\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="1c9fab32-f50d-4b18-a4ab-ab3066853008" pod="opendatahub/test-curl-7281-140466090978448" Jul 30 21:15:42.129706 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:42.129635 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9fab32-f50d-4b18-a4ab-ab3066853008" path="/var/lib/kubelet/pods/1c9fab32-f50d-4b18-a4ab-ab3066853008/volumes" Jul 30 21:15:42.879635 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:42.879595 2537 generic.go:356] "Generic (PLEG): container finished" podID="e79e2260-07f8-4075-949f-f377dab31998" containerID="2603c175501a3148bdf9ba4a64848151c2083b6731fdda8bb7293010a3904f86" exitCode=0 Jul 30 21:15:42.879811 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:42.879676 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466091077680" event={"ID":"e79e2260-07f8-4075-949f-f377dab31998","Type":"ContainerDied","Data":"2603c175501a3148bdf9ba4a64848151c2083b6731fdda8bb7293010a3904f86"} Jul 30 21:15:43.407937 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.407895 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" podUID="7e7fda69-87c2-4bdd-a359-83262a4eb9a4" containerName="main" containerID="cri-o://1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d" gracePeriod=45 Jul 30 21:15:43.670399 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.670336 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:15:43.757459 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757423 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-model-cache\") pod \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " Jul 30 21:15:43.757642 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757483 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-home\") pod \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " Jul 30 21:15:43.757642 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757516 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tmp-dir\") pod \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " Jul 30 21:15:43.757642 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757542 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knch5\" (UniqueName: \"kubernetes.io/projected/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-kube-api-access-knch5\") pod \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " Jul 30 21:15:43.757642 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757584 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tls-certs\") pod \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " Jul 30 21:15:43.757870 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757643 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e7fda69-87c2-4bdd-a359-83262a4eb9a4" (UID: "7e7fda69-87c2-4bdd-a359-83262a4eb9a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:15:43.757870 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757762 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e7fda69-87c2-4bdd-a359-83262a4eb9a4" (UID: "7e7fda69-87c2-4bdd-a359-83262a4eb9a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:15:43.757870 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757770 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-home" (OuterVolumeSpecName: "home") pod "7e7fda69-87c2-4bdd-a359-83262a4eb9a4" (UID: "7e7fda69-87c2-4bdd-a359-83262a4eb9a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:15:43.758032 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757909 2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-model-cache\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:43.758032 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757928 2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-home\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:43.758032 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.757942 2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tmp-dir\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:43.759725 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.759695 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e7fda69-87c2-4bdd-a359-83262a4eb9a4" (UID: "7e7fda69-87c2-4bdd-a359-83262a4eb9a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 21:15:43.759725 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.759712 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-kube-api-access-knch5" (OuterVolumeSpecName: "kube-api-access-knch5") pod "7e7fda69-87c2-4bdd-a359-83262a4eb9a4" (UID: "7e7fda69-87c2-4bdd-a359-83262a4eb9a4"). InnerVolumeSpecName "kube-api-access-knch5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:15:43.858726 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.858692 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-dshm\") pod \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\" (UID: \"7e7fda69-87c2-4bdd-a359-83262a4eb9a4\") " Jul 30 21:15:43.859025 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.859003 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knch5\" (UniqueName: \"kubernetes.io/projected/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-kube-api-access-knch5\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:43.859075 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.859032 2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-tls-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:43.860757 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.860737 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-dshm" (OuterVolumeSpecName: "dshm") pod "7e7fda69-87c2-4bdd-a359-83262a4eb9a4" (UID: "7e7fda69-87c2-4bdd-a359-83262a4eb9a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:15:43.885953 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.885924 2537 generic.go:356] "Generic (PLEG): container finished" podID="7e7fda69-87c2-4bdd-a359-83262a4eb9a4" containerID="1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d" exitCode=0 Jul 30 21:15:43.886093 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.885969 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" Jul 30 21:15:43.886093 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.886012 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" event={"ID":"7e7fda69-87c2-4bdd-a359-83262a4eb9a4","Type":"ContainerDied","Data":"1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d"} Jul 30 21:15:43.886093 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.886060 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf" event={"ID":"7e7fda69-87c2-4bdd-a359-83262a4eb9a4","Type":"ContainerDied","Data":"edeec6fd41c71544bff748053a8e99b8d9bd84deccf495b8ec175ca82292fa5f"} Jul 30 21:15:43.886093 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.886082 2537 scope.go:117] "RemoveContainer" containerID="1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d" Jul 30 21:15:43.895086 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.895069 2537 scope.go:117] "RemoveContainer" containerID="1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d" Jul 30 21:15:43.895342 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:15:43.895322 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d\": container with ID starting with 1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d not found: ID does not exist" containerID="1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d" Jul 30 21:15:43.895417 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.895355 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d"} err="failed to get container status \"1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d\": rpc error: code = NotFound desc = could not find container \"1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d\": container with ID starting with 1eb84237b7037ae0f85f38c9fbddab3dd965516b69cac9c10e2b680099a6938d not found: ID does not exist" Jul 30 21:15:43.907954 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.907930 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf"] Jul 30 21:15:43.913397 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.913352 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e226df/rate-test-model-e226df-kserve-6b7949976c-9btqf"] Jul 30 21:15:43.960552 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:43.960525 2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e7fda69-87c2-4bdd-a359-83262a4eb9a4-dshm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:44.027726 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.027703 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:44.047181 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.047155 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7281-140466091077680_e79e2260-07f8-4075-949f-f377dab31998/test-curl-7281-140466091077680/0.log" Jul 30 21:15:44.061389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.061366 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq6xg\" (UniqueName: \"kubernetes.io/projected/e79e2260-07f8-4075-949f-f377dab31998-kube-api-access-wq6xg\") pod \"e79e2260-07f8-4075-949f-f377dab31998\" (UID: \"e79e2260-07f8-4075-949f-f377dab31998\") " Jul 30 21:15:44.063366 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.063347 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79e2260-07f8-4075-949f-f377dab31998-kube-api-access-wq6xg" (OuterVolumeSpecName: "kube-api-access-wq6xg") pod "e79e2260-07f8-4075-949f-f377dab31998" (UID: "e79e2260-07f8-4075-949f-f377dab31998"). InnerVolumeSpecName "kube-api-access-wq6xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:15:44.072991 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.072965 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:44.076393 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.076374 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:44.130876 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.130846 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e7fda69-87c2-4bdd-a359-83262a4eb9a4" path="/var/lib/kubelet/pods/7e7fda69-87c2-4bdd-a359-83262a4eb9a4/volumes" Jul 30 21:15:44.131180 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.131167 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e79e2260-07f8-4075-949f-f377dab31998" path="/var/lib/kubelet/pods/e79e2260-07f8-4075-949f-f377dab31998/volumes" Jul 30 21:15:44.162702 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.162669 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq6xg\" (UniqueName: \"kubernetes.io/projected/e79e2260-07f8-4075-949f-f377dab31998-kube-api-access-wq6xg\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:44.200585 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.200508 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:44.200864 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.200849 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e79e2260-07f8-4075-949f-f377dab31998" containerName="test-curl-7281-140466091077680" Jul 30 21:15:44.200910 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.200866 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="e79e2260-07f8-4075-949f-f377dab31998" containerName="test-curl-7281-140466091077680" Jul 30 21:15:44.200910 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.200890 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7fda69-87c2-4bdd-a359-83262a4eb9a4" containerName="main" Jul 30 21:15:44.200910 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.200895 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7fda69-87c2-4bdd-a359-83262a4eb9a4" containerName="main" Jul 30 21:15:44.201013 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.200960 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e7fda69-87c2-4bdd-a359-83262a4eb9a4" containerName="main" Jul 30 21:15:44.201013 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.200966 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="e79e2260-07f8-4075-949f-f377dab31998" containerName="test-curl-7281-140466091077680" Jul 30 21:15:44.203975 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.203958 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:44.210500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.210477 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:44.263570 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.263546 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/a62388a1-5503-4077-a6fe-0272e2d2baf9-kube-api-access-89pkm\") pod \"test-curl-7281-140466091077680\" (UID: \"a62388a1-5503-4077-a6fe-0272e2d2baf9\") " pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:44.364934 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.364903 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/a62388a1-5503-4077-a6fe-0272e2d2baf9-kube-api-access-89pkm\") pod \"test-curl-7281-140466091077680\" (UID: \"a62388a1-5503-4077-a6fe-0272e2d2baf9\") " pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:44.373832 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.373804 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/a62388a1-5503-4077-a6fe-0272e2d2baf9-kube-api-access-89pkm\") pod \"test-curl-7281-140466091077680\" (UID: \"a62388a1-5503-4077-a6fe-0272e2d2baf9\") " pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:44.513980 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.513905 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:44.640837 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.640815 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:44.642862 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:15:44.642832 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62388a1_5503_4077_a6fe_0272e2d2baf9.slice/crio-0cad4fbca14d18657a2d2c862286f4297b70b0685ad4bb5470d66e5a1ed2bd19 WatchSource:0}: Error finding container 0cad4fbca14d18657a2d2c862286f4297b70b0685ad4bb5470d66e5a1ed2bd19: Status 404 returned error can't find the container with id 0cad4fbca14d18657a2d2c862286f4297b70b0685ad4bb5470d66e5a1ed2bd19 Jul 30 21:15:44.896649 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.896617 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:44.896836 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.896656 2537 scope.go:117] "RemoveContainer" containerID="2603c175501a3148bdf9ba4a64848151c2083b6731fdda8bb7293010a3904f86" Jul 30 21:15:44.897829 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:44.897802 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466091077680" event={"ID":"a62388a1-5503-4077-a6fe-0272e2d2baf9","Type":"ContainerStarted","Data":"0cad4fbca14d18657a2d2c862286f4297b70b0685ad4bb5470d66e5a1ed2bd19"} Jul 30 21:15:45.905495 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:45.905457 2537 generic.go:356] "Generic (PLEG): container finished" podID="a62388a1-5503-4077-a6fe-0272e2d2baf9" containerID="a6a4b66cd7035c8a699358fd0f6dddf97853783fbe3217d322f58c3a102030f9" exitCode=0 Jul 30 21:15:45.905920 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:45.905547 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7281-140466091077680" event={"ID":"a62388a1-5503-4077-a6fe-0272e2d2baf9","Type":"ContainerDied","Data":"a6a4b66cd7035c8a699358fd0f6dddf97853783fbe3217d322f58c3a102030f9"} Jul 30 21:15:47.045031 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.045009 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:47.066662 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.066639 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7281-140466091077680_a62388a1-5503-4077-a6fe-0272e2d2baf9/test-curl-7281-140466091077680/0.log" Jul 30 21:15:47.085955 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.085920 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/a62388a1-5503-4077-a6fe-0272e2d2baf9-kube-api-access-89pkm\") pod \"a62388a1-5503-4077-a6fe-0272e2d2baf9\" (UID: \"a62388a1-5503-4077-a6fe-0272e2d2baf9\") " Jul 30 21:15:47.087947 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.087924 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62388a1-5503-4077-a6fe-0272e2d2baf9-kube-api-access-89pkm" (OuterVolumeSpecName: "kube-api-access-89pkm") pod "a62388a1-5503-4077-a6fe-0272e2d2baf9" (UID: "a62388a1-5503-4077-a6fe-0272e2d2baf9"). InnerVolumeSpecName "kube-api-access-89pkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:15:47.091854 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.091828 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:47.095654 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.095633 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7281-140466091077680"] Jul 30 21:15:47.186654 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.186569 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pkm\" (UniqueName: \"kubernetes.io/projected/a62388a1-5503-4077-a6fe-0272e2d2baf9-kube-api-access-89pkm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:15:47.920013 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.919988 2537 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cad4fbca14d18657a2d2c862286f4297b70b0685ad4bb5470d66e5a1ed2bd19" Jul 30 21:15:47.920013 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:47.920005 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:47.932009 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:15:47.931981 2537 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7281-140466091077680\" is forbidden: User \"system:node:ip-10-0-131-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-223.ec2.internal' and this object" podUID="a62388a1-5503-4077-a6fe-0272e2d2baf9" pod="opendatahub/test-curl-7281-140466091077680" Jul 30 21:15:48.129415 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:48.129361 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62388a1-5503-4077-a6fe-0272e2d2baf9" path="/var/lib/kubelet/pods/a62388a1-5503-4077-a6fe-0272e2d2baf9/volumes" Jul 30 21:15:48.735319 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:48.735274 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:48.795444 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:48.795412 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:50.781320 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:50.781288 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:50.878884 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:50.878850 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:51.914030 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:51.913999 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:52.190138 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:52.190044 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:52.555407 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:52.555296 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:52.728962 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:52.728923 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:15:54.952314 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:15:54.952286 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:16:15.840743 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:15.840705 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:16.524580 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.524525 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:16.928990 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.928953 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz"] Jul 30 21:16:16.929435 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.929308 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62388a1-5503-4077-a6fe-0272e2d2baf9" containerName="test-curl-7281-140466091077680" Jul 30 21:16:16.929435 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.929321 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62388a1-5503-4077-a6fe-0272e2d2baf9" containerName="test-curl-7281-140466091077680" Jul 30 21:16:16.929435 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.929390 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62388a1-5503-4077-a6fe-0272e2d2baf9" containerName="test-curl-7281-140466091077680" Jul 30 21:16:16.934486 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.934464 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:16.937151 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.937127 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6967f3-openshift-default-dockercfg-w25h2\"" Jul 30 21:16:16.948313 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.948290 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz"] Jul 30 21:16:16.973012 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:16.972977 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:17.042983 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.042941 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-socket\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.042983 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.042983 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-envoy\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.043212 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.043009 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zs5k\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-kube-api-access-9zs5k\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.043212 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.043090 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-certs\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.043212 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.043157 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd58a576-d8cc-4953-9452-8f3ec692ee86-istiod-ca-cert\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.043389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.043244 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-credential-socket\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.043389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.043280 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-data\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.043389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.043312 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-token\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.043389 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.043340 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-podinfo\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144398 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144362 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd58a576-d8cc-4953-9452-8f3ec692ee86-istiod-ca-cert\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144591 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144403 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-credential-socket\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144591 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144535 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-data\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144591 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144585 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-token\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144761 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144623 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-podinfo\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144761 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144697 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-socket\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144761 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144727 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-envoy\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144944 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144759 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zs5k\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-kube-api-access-9zs5k\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144944 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144765 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-credential-socket\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144944 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144843 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-certs\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.144944 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.144891 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-data\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.145162 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.145108 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-socket\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.145162 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.145134 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-certs\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.145393 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.145377 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd58a576-d8cc-4953-9452-8f3ec692ee86-istiod-ca-cert\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.147091 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.147072 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-envoy\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.147181 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.147163 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-podinfo\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.155456 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.155429 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-token\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.155571 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.155552 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zs5k\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-kube-api-access-9zs5k\") pod \"e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.238086 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.237995 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:17.245019 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.244988 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:17.392163 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.392139 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz"] Jul 30 21:16:17.394355 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:16:17.394328 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd58a576_d8cc_4953_9452_8f3ec692ee86.slice/crio-bfd1e916d22666af52831121b9c64863333ee98230b1591a0dee3d3319334b8a WatchSource:0}: Error finding container bfd1e916d22666af52831121b9c64863333ee98230b1591a0dee3d3319334b8a: Status 404 returned error can't find the container with id bfd1e916d22666af52831121b9c64863333ee98230b1591a0dee3d3319334b8a Jul 30 21:16:17.396471 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.396436 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:16:17.396558 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.396503 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:16:17.396558 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:17.396531 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:16:18.059304 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:18.059265 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" event={"ID":"dd58a576-d8cc-4953-9452-8f3ec692ee86","Type":"ContainerStarted","Data":"60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234"} Jul 30 21:16:18.059811 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:18.059318 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" event={"ID":"dd58a576-d8cc-4953-9452-8f3ec692ee86","Type":"ContainerStarted","Data":"bfd1e916d22666af52831121b9c64863333ee98230b1591a0dee3d3319334b8a"} Jul 30 21:16:18.080877 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:18.080828 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podStartSLOduration=2.080813982 podStartE2EDuration="2.080813982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:16:18.079146282 +0000 UTC m=+3304.339458830" watchObservedRunningTime="2026-07-30 21:16:18.080813982 +0000 UTC m=+3304.341126509" Jul 30 21:16:18.245929 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:18.245900 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:18.250918 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:18.250895 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:19.063993 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:19.063961 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:19.064929 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:19.064911 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:20.037242 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:20.037207 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:20.097317 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:20.097284 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:21.145095 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:21.145055 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:21.252891 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:21.252835 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:21.648276 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:21.648242 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:21.747198 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:21.747157 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:16:25.164975 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:25.164941 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:16:34.314666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:34.314063 2537 patch_prober.go:28] interesting pod/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": read tcp 10.134.0.2:41236->10.134.0.49:15021: read: connection reset by peer" start-of-body= Jul 30 21:16:34.314666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:34.314164 2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": read tcp 10.134.0.2:41236->10.134.0.49:15021: read: connection reset by peer" Jul 30 21:16:35.150324 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:35.150284 2537 generic.go:356] "Generic (PLEG): container finished" podID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerID="60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234" exitCode=0 Jul 30 21:16:35.150502 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:35.150357 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" event={"ID":"dd58a576-d8cc-4953-9452-8f3ec692ee86","Type":"ContainerDied","Data":"60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234"} Jul 30 21:16:35.150729 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:35.150715 2537 scope.go:117] "RemoveContainer" containerID="60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234" Jul 30 21:16:35.151717 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:35.151690 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:16:35.151764 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:35.151757 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:16:35.151825 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:35.151811 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:16:36.156284 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:36.156207 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" event={"ID":"dd58a576-d8cc-4953-9452-8f3ec692ee86","Type":"ContainerStarted","Data":"6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1"} Jul 30 21:16:36.245583 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:36.245545 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:36.245832 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:36.245807 2537 patch_prober.go:28] interesting pod/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 30 21:16:36.245894 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:36.245875 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 30 21:16:37.245709 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:37.245674 2537 patch_prober.go:28] interesting pod/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 30 21:16:37.246112 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:37.245726 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 30 21:16:38.245887 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:38.245846 2537 patch_prober.go:28] interesting pod/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 30 21:16:38.246289 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:38.245906 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 30 21:16:39.245919 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:39.245881 2537 patch_prober.go:28] interesting pod/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 30 21:16:39.246323 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:39.245933 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 30 21:16:40.245526 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:40.245480 2537 patch_prober.go:28] interesting pod/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 30 21:16:40.245707 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:40.245550 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 30 21:16:41.249820 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:41.249788 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:41.250188 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:41.250146 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:41.250878 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:41.250859 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:16:44.995300 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:16:44.995267 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:04.766194 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.766151 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q"] Jul 30 21:17:04.769523 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.769501 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.773140 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.773121 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-720e63\"/\"kube-root-ca.crt\"" Jul 30 21:17:04.773270 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.773179 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-720e63\"/\"ipp-route-720e63-kserve-self-signed-certs\"" Jul 30 21:17:04.773270 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.773239 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-720e63\"/\"default-dockercfg-n55lr\"" Jul 30 21:17:04.773270 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.773253 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-720e63\"/\"openshift-service-ca.crt\"" Jul 30 21:17:04.779173 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.779152 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q"] Jul 30 21:17:04.852162 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.852129 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:04.870915 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.870889 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-model-cache\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.870999 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.870926 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7lf\" (UniqueName: \"kubernetes.io/projected/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-kube-api-access-vj7lf\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.871071 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.871053 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tls-certs\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.871118 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.871104 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-dshm\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.871155 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.871146 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-home\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.871192 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.871167 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tmp-dir\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972454 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972423 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tls-certs\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972647 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972468 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-dshm\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972647 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972484 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-home\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972647 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972501 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tmp-dir\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972647 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972532 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-model-cache\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972647 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972553 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7lf\" (UniqueName: \"kubernetes.io/projected/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-kube-api-access-vj7lf\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972944 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972854 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-home\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.972989 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.972948 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tmp-dir\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.973028 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.973007 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-model-cache\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.974899 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.974876 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-dshm\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.975140 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.975123 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tls-certs\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:04.980273 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:04.980252 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7lf\" (UniqueName: \"kubernetes.io/projected/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-kube-api-access-vj7lf\") pod \"ipp-route-720e63-kserve-8586d699dc-k8g6q\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:05.079240 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:05.079150 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:05.220833 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:05.220789 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q"] Jul 30 21:17:05.224662 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:17:05.224624 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb59e3f0e_3bcb_4839_89fd_ff2605046ec0.slice/crio-a6d3b5b9c73612fd9a8fd87f3a04fed4400d8fda8d48ef684f97a599e6cfd5b6 WatchSource:0}: Error finding container a6d3b5b9c73612fd9a8fd87f3a04fed4400d8fda8d48ef684f97a599e6cfd5b6: Status 404 returned error can't find the container with id a6d3b5b9c73612fd9a8fd87f3a04fed4400d8fda8d48ef684f97a599e6cfd5b6 Jul 30 21:17:05.293796 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:05.293748 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" event={"ID":"b59e3f0e-3bcb-4839-89fd-ff2605046ec0","Type":"ContainerStarted","Data":"08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab"} Jul 30 21:17:05.293901 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:05.293812 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" event={"ID":"b59e3f0e-3bcb-4839-89fd-ff2605046ec0","Type":"ContainerStarted","Data":"a6d3b5b9c73612fd9a8fd87f3a04fed4400d8fda8d48ef684f97a599e6cfd5b6"} Jul 30 21:17:05.310555 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:05.310505 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" podStartSLOduration=1.310489343 podStartE2EDuration="1.310489343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:17:05.310022114 +0000 UTC m=+3351.570334639" watchObservedRunningTime="2026-07-30 21:17:05.310489343 +0000 UTC m=+3351.570801869" Jul 30 21:17:07.750640 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:07.750610 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:17:15.079499 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:15.079464 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:15.079929 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:15.079606 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:15.092255 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:15.092230 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:15.352805 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:15.352714 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:17:30.951909 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:30.951871 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:17:42.621290 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:42.621254 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:42.928694 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:42.928618 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622"] Jul 30 21:17:42.932256 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:42.932234 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:42.934962 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:42.934943 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e026c1-openshift-default-dockercfg-vqnlh\"" Jul 30 21:17:42.944342 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:42.944314 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622"] Jul 30 21:17:42.952190 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:42.952154 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:43.000530 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000481 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-socket\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000530 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000526 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000816 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000549 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-envoy\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000816 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000753 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-credential-socket\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000940 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000824 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-data\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000940 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000850 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8p8h\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-kube-api-access-c8p8h\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000940 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000879 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-podinfo\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000940 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000902 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-token\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.000940 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.000926 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-certs\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102075 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102021 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-socket\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102075 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102075 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102109 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-envoy\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102230 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-credential-socket\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102255 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-data\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102281 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8p8h\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-kube-api-access-c8p8h\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102309 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-podinfo\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102604 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102333 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-token\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102604 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102361 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-certs\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102604 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102498 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-socket\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.102762 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.102661 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-certs\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.103104 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.103081 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-credential-socket\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.103280 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.103213 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-data\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.103421 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.103373 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.104836 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.104815 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-envoy\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.105836 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.105816 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-podinfo\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.111676 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.111655 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-token\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.112352 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.112332 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8p8h\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-kube-api-access-c8p8h\") pod \"e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.178766 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.178663 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:43.243690 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.243656 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:43.386037 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.385907 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622"] Jul 30 21:17:43.388992 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:17:43.388960 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb663b0c9_0663_4bab_9cae_ba8c3d322bcd.slice/crio-1452a1795dd3db79aa0326481c9d81fa49acd4dab618ee55618c88a4848ed476 WatchSource:0}: Error finding container 1452a1795dd3db79aa0326481c9d81fa49acd4dab618ee55618c88a4848ed476: Status 404 returned error can't find the container with id 1452a1795dd3db79aa0326481c9d81fa49acd4dab618ee55618c88a4848ed476 Jul 30 21:17:43.391512 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.391480 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:17:43.391578 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.391547 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:17:43.391617 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.391575 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:17:43.471477 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.471436 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" event={"ID":"b663b0c9-0663-4bab-9cae-ba8c3d322bcd","Type":"ContainerStarted","Data":"880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8"} Jul 30 21:17:43.471633 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.471507 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" event={"ID":"b663b0c9-0663-4bab-9cae-ba8c3d322bcd","Type":"ContainerStarted","Data":"1452a1795dd3db79aa0326481c9d81fa49acd4dab618ee55618c88a4848ed476"} Jul 30 21:17:43.491065 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:43.491001 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podStartSLOduration=1.490977792 podStartE2EDuration="1.490977792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:17:43.490050587 +0000 UTC m=+3389.750363141" watchObservedRunningTime="2026-07-30 21:17:43.490977792 +0000 UTC m=+3389.751290320" Jul 30 21:17:44.244438 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:44.244404 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:44.249563 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:44.249540 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:44.479593 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:44.479559 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:44.480457 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:44.480437 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:17:44.652241 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:44.652210 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:44.709285 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:44.709250 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:45.423231 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:45.423199 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:45.520673 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:45.520635 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:46.632422 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:46.632388 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:46.802811 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:46.797650 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:47.162363 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:47.162323 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:47.266190 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:47.266158 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:17:51.852401 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:17:51.852369 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:18:00.521942 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:00.521894 2537 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 21:18:00.522426 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:00.521969 2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 21:18:01.567007 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:01.566971 2537 generic.go:356] "Generic (PLEG): container finished" podID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerID="880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8" exitCode=0 Jul 30 21:18:01.567408 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:01.567043 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" event={"ID":"b663b0c9-0663-4bab-9cae-ba8c3d322bcd","Type":"ContainerDied","Data":"880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8"} Jul 30 21:18:01.567450 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:01.567443 2537 scope.go:117] "RemoveContainer" containerID="880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8" Jul 30 21:18:01.568546 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:01.568519 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:18:01.568600 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:01.568584 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:18:01.568635 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:01.568615 2537 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 21:18:02.573654 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:02.573613 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" event={"ID":"b663b0c9-0663-4bab-9cae-ba8c3d322bcd","Type":"ContainerStarted","Data":"722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d"} Jul 30 21:18:03.243850 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:03.243808 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:18:03.244140 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:03.244114 2537 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 30 21:18:03.244201 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:03.244179 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 30 21:18:04.244737 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:04.244692 2537 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 30 21:18:04.245171 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:04.244756 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 30 21:18:05.244835 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:05.244765 2537 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 30 21:18:05.245244 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:05.244861 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 30 21:18:06.244506 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:06.244415 2537 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 30 21:18:06.244506 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:06.244479 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 30 21:18:07.244145 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:07.244096 2537 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 30 21:18:07.244545 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:07.244184 2537 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 30 21:18:08.248191 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:08.248160 2537 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:18:08.248654 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:08.248446 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:18:08.249125 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:08.249105 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:18:10.459005 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:10.458970 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:17.321737 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:17.321699 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:17.388085 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:17.388039 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:17.486568 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:17.486535 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:17.881206 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:17.881171 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:20.561815 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:20.561757 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:18:21.439844 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:21.439756 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:23.643827 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:23.643763 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:23.717788 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:23.717740 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:23.887877 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:23.885193 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622"] Jul 30 21:18:23.887877 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:23.885617 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" containerID="cri-o://722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d" gracePeriod=30 Jul 30 21:18:23.940873 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:23.940767 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:24.147949 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:24.147907 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:29.138651 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.138627 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:18:29.235268 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235189 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-socket\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235268 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235231 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-token\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235298 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-data\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235327 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-credential-socket\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235357 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istiod-ca-cert\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235389 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-envoy\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235440 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-certs\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235500 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235472 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8p8h\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-kube-api-access-c8p8h\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235840 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235517 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-podinfo\") pod \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\" (UID: \"b663b0c9-0663-4bab-9cae-ba8c3d322bcd\") " Jul 30 21:18:29.235840 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235511 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:18:29.235840 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235625 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-data" (OuterVolumeSpecName: "istio-data") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:18:29.235840 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235640 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:18:29.235840 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235738 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:18:29.236093 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.235800 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:18:29.237691 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.237660 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-token" (OuterVolumeSpecName: "istio-token") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:18:29.237821 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.237804 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:18:29.237991 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.237973 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-kube-api-access-c8p8h" (OuterVolumeSpecName: "kube-api-access-c8p8h") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "kube-api-access-c8p8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:18:29.238067 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.238028 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b663b0c9-0663-4bab-9cae-ba8c3d322bcd" (UID: "b663b0c9-0663-4bab-9cae-ba8c3d322bcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 21:18:29.336550 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336513 2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-socket\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336550 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336542 2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-token\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336550 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336553 2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-data\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336804 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336561 2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-credential-socket\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336804 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336570 2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istiod-ca-cert\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336804 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336578 2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-envoy\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336804 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336587 2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-workload-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336804 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336595 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8p8h\" (UniqueName: \"kubernetes.io/projected/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-kube-api-access-c8p8h\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.336804 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.336604 2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b663b0c9-0663-4bab-9cae-ba8c3d322bcd-istio-podinfo\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:18:29.712313 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.712274 2537 generic.go:356] "Generic (PLEG): container finished" podID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerID="722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d" exitCode=0 Jul 30 21:18:29.712509 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.712321 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" Jul 30 21:18:29.712509 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.712358 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" event={"ID":"b663b0c9-0663-4bab-9cae-ba8c3d322bcd","Type":"ContainerDied","Data":"722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d"} Jul 30 21:18:29.712509 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.712411 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622" event={"ID":"b663b0c9-0663-4bab-9cae-ba8c3d322bcd","Type":"ContainerDied","Data":"1452a1795dd3db79aa0326481c9d81fa49acd4dab618ee55618c88a4848ed476"} Jul 30 21:18:29.712509 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.712426 2537 scope.go:117] "RemoveContainer" containerID="722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d" Jul 30 21:18:29.728302 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.728281 2537 scope.go:117] "RemoveContainer" containerID="880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8" Jul 30 21:18:29.736321 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.736302 2537 scope.go:117] "RemoveContainer" containerID="722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d" Jul 30 21:18:29.736623 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:18:29.736597 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d\": container with ID starting with 722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d not found: ID does not exist" containerID="722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d" Jul 30 21:18:29.736730 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.736628 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d"} err="failed to get container status \"722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d\": rpc error: code = NotFound desc = could not find container \"722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d\": container with ID starting with 722905acbf32e7700bdaad6fca4894d8f08287805128f08cca8109e8cd7a660d not found: ID does not exist" Jul 30 21:18:29.736730 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.736652 2537 scope.go:117] "RemoveContainer" containerID="880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8" Jul 30 21:18:29.737062 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:18:29.737041 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8\": container with ID starting with 880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8 not found: ID does not exist" containerID="880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8" Jul 30 21:18:29.737165 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.737067 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8"} err="failed to get container status \"880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8\": rpc error: code = NotFound desc = could not find container \"880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8\": container with ID starting with 880338b590955c0e08c5c14be0a06699746f7c5f5fe80a5d51ec72e3ba5c13c8 not found: ID does not exist" Jul 30 21:18:29.738216 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.738196 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622"] Jul 30 21:18:29.742344 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.742323 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e026c1-openshift-default-6f5bcb76d6-6w622"] Jul 30 21:18:29.963295 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:29.963202 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:30.018143 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:30.018106 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:30.129666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:30.129631 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" path="/var/lib/kubelet/pods/b663b0c9-0663-4bab-9cae-ba8c3d322bcd/volumes" Jul 30 21:18:40.958037 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:40.958005 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:18:43.244135 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:43.244097 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:43.344063 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:43.344028 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:43.414233 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:43.414191 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:43.492195 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:43.492154 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:43.788894 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:43.788853 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:48.774430 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:48.774398 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q"] Jul 30 21:18:48.871917 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:48.871871 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:18:50.354644 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:50.354607 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:18:58.320352 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:18:58.320321 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:03.793478 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:03.793399 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" podUID="b59e3f0e-3bcb-4839-89fd-ff2605046ec0" containerName="main" containerID="cri-o://08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab" gracePeriod=45 Jul 30 21:19:04.048064 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.048010 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:19:04.141511 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141482 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tls-certs\") pod \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " Jul 30 21:19:04.141666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141522 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-dshm\") pod \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " Jul 30 21:19:04.141666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141548 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-model-cache\") pod \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " Jul 30 21:19:04.141666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141572 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj7lf\" (UniqueName: \"kubernetes.io/projected/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-kube-api-access-vj7lf\") pod \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " Jul 30 21:19:04.141666 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141643 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-home\") pod \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " Jul 30 21:19:04.141925 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141692 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tmp-dir\") pod \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\" (UID: \"b59e3f0e-3bcb-4839-89fd-ff2605046ec0\") " Jul 30 21:19:04.141925 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141871 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b59e3f0e-3bcb-4839-89fd-ff2605046ec0" (UID: "b59e3f0e-3bcb-4839-89fd-ff2605046ec0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:19:04.142027 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141989 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-home" (OuterVolumeSpecName: "home") pod "b59e3f0e-3bcb-4839-89fd-ff2605046ec0" (UID: "b59e3f0e-3bcb-4839-89fd-ff2605046ec0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:19:04.142027 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.141994 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b59e3f0e-3bcb-4839-89fd-ff2605046ec0" (UID: "b59e3f0e-3bcb-4839-89fd-ff2605046ec0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:19:04.142110 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.142060 2537 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-model-cache\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:19:04.143676 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.143650 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-dshm" (OuterVolumeSpecName: "dshm") pod "b59e3f0e-3bcb-4839-89fd-ff2605046ec0" (UID: "b59e3f0e-3bcb-4839-89fd-ff2605046ec0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:19:04.144100 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.144077 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b59e3f0e-3bcb-4839-89fd-ff2605046ec0" (UID: "b59e3f0e-3bcb-4839-89fd-ff2605046ec0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 21:19:04.144151 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.144090 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-kube-api-access-vj7lf" (OuterVolumeSpecName: "kube-api-access-vj7lf") pod "b59e3f0e-3bcb-4839-89fd-ff2605046ec0" (UID: "b59e3f0e-3bcb-4839-89fd-ff2605046ec0"). InnerVolumeSpecName "kube-api-access-vj7lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:19:04.243375 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.243340 2537 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tmp-dir\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:19:04.243375 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.243369 2537 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-tls-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:19:04.243375 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.243377 2537 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-dshm\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:19:04.243609 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.243387 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7lf\" (UniqueName: \"kubernetes.io/projected/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-kube-api-access-vj7lf\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:19:04.243609 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.243396 2537 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b59e3f0e-3bcb-4839-89fd-ff2605046ec0-home\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:19:04.885044 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.885012 2537 generic.go:356] "Generic (PLEG): container finished" podID="b59e3f0e-3bcb-4839-89fd-ff2605046ec0" containerID="08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab" exitCode=0 Jul 30 21:19:04.885432 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.885057 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" Jul 30 21:19:04.885432 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.885099 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" event={"ID":"b59e3f0e-3bcb-4839-89fd-ff2605046ec0","Type":"ContainerDied","Data":"08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab"} Jul 30 21:19:04.885432 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.885141 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q" event={"ID":"b59e3f0e-3bcb-4839-89fd-ff2605046ec0","Type":"ContainerDied","Data":"a6d3b5b9c73612fd9a8fd87f3a04fed4400d8fda8d48ef684f97a599e6cfd5b6"} Jul 30 21:19:04.885432 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.885166 2537 scope.go:117] "RemoveContainer" containerID="08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab" Jul 30 21:19:04.894350 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.894325 2537 scope.go:117] "RemoveContainer" containerID="08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab" Jul 30 21:19:04.894630 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:19:04.894609 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab\": container with ID starting with 08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab not found: ID does not exist" containerID="08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab" Jul 30 21:19:04.894675 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.894639 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab"} err="failed to get container status \"08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab\": rpc error: code = NotFound desc = could not find container \"08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab\": container with ID starting with 08357e5338b1ab29a45bc8346a3435e45b3a1efab2b26b7273a1917f77ea1bab not found: ID does not exist" Jul 30 21:19:04.907142 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.907117 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q"] Jul 30 21:19:04.911132 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:04.911111 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-720e63/ipp-route-720e63-kserve-8586d699dc-k8g6q"] Jul 30 21:19:06.130194 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:06.130154 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59e3f0e-3bcb-4839-89fd-ff2605046ec0" path="/var/lib/kubelet/pods/b59e3f0e-3bcb-4839-89fd-ff2605046ec0/volumes" Jul 30 21:19:25.502477 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:25.502430 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:25.768184 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:25.768105 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:31.462998 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:31.462916 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:37.290364 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:37.290286 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:37.390002 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:37.389963 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:37.576930 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:37.576849 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:37.811203 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:37.811163 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:48.504340 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:48.504309 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:48.599792 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:48.599747 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:48.699033 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:48.698995 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:48.784658 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:48.784565 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:49.129994 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:49.129961 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:50.849912 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:50.849876 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:19:54.874446 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:54.874411 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:54.974870 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:54.974831 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:55.126027 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:55.125938 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz"] Jul 30 21:19:55.126197 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:55.126100 2537 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" containerID="cri-o://6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1" gracePeriod=30 Jul 30 21:19:55.154773 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:55.154748 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:55.373798 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:55.373747 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:19:56.253238 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:56.253199 2537 patch_prober.go:28] interesting pod/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 21:19:56.253665 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:19:56.253270 2537 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 21:20:00.369758 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.369730 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:20:00.421059 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.420992 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-envoy\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421200 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421071 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-socket\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421200 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421087 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-data\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421200 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421105 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-token\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421200 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421130 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-certs\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421200 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421146 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-credential-socket\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421200 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421164 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-podinfo\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421200 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421195 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd58a576-d8cc-4953-9452-8f3ec692ee86-istiod-ca-cert\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421596 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421242 2537 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zs5k\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-kube-api-access-9zs5k\") pod \"dd58a576-d8cc-4953-9452-8f3ec692ee86\" (UID: \"dd58a576-d8cc-4953-9452-8f3ec692ee86\") " Jul 30 21:20:00.421596 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421364 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:20:00.421596 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421437 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:20:00.421596 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421566 2537 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-socket\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.421596 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421584 2537 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-data\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.421887 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421636 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:20:00.421887 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421584 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:20:00.421887 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.421747 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd58a576-d8cc-4953-9452-8f3ec692ee86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 21:20:00.423376 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.423349 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 21:20:00.423473 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.423353 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 21:20:00.423473 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.423374 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:20:00.423473 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.423417 2537 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-kube-api-access-9zs5k" (OuterVolumeSpecName: "kube-api-access-9zs5k") pod "dd58a576-d8cc-4953-9452-8f3ec692ee86" (UID: "dd58a576-d8cc-4953-9452-8f3ec692ee86"). InnerVolumeSpecName "kube-api-access-9zs5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 21:20:00.522524 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.522486 2537 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-token\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.522524 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.522518 2537 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-workload-certs\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.522524 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.522528 2537 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-credential-socket\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.522763 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.522536 2537 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-podinfo\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.522763 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.522545 2537 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd58a576-d8cc-4953-9452-8f3ec692ee86-istiod-ca-cert\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.522763 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.522553 2537 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zs5k\" (UniqueName: \"kubernetes.io/projected/dd58a576-d8cc-4953-9452-8f3ec692ee86-kube-api-access-9zs5k\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:00.522763 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:00.522561 2537 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd58a576-d8cc-4953-9452-8f3ec692ee86-istio-envoy\") on node \"ip-10-0-131-223.ec2.internal\" DevicePath \"\"" Jul 30 21:20:01.158979 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.158945 2537 generic.go:356] "Generic (PLEG): container finished" podID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerID="6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1" exitCode=0 Jul 30 21:20:01.159151 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.158993 2537 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" Jul 30 21:20:01.159151 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.158998 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" event={"ID":"dd58a576-d8cc-4953-9452-8f3ec692ee86","Type":"ContainerDied","Data":"6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1"} Jul 30 21:20:01.159151 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.159047 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz" event={"ID":"dd58a576-d8cc-4953-9452-8f3ec692ee86","Type":"ContainerDied","Data":"bfd1e916d22666af52831121b9c64863333ee98230b1591a0dee3d3319334b8a"} Jul 30 21:20:01.159151 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.159068 2537 scope.go:117] "RemoveContainer" containerID="6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1" Jul 30 21:20:01.167753 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.167731 2537 scope.go:117] "RemoveContainer" containerID="60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234" Jul 30 21:20:01.175312 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.175291 2537 scope.go:117] "RemoveContainer" containerID="6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1" Jul 30 21:20:01.175520 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:20:01.175496 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1\": container with ID starting with 6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1 not found: ID does not exist" containerID="6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1" Jul 30 21:20:01.175564 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.175529 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1"} err="failed to get container status \"6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1\": rpc error: code = NotFound desc = could not find container \"6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1\": container with ID starting with 6a09fd1a58db463c47b7ff06878d7c3f575e4fd9b181f2147f3bc80882c657b1 not found: ID does not exist" Jul 30 21:20:01.175564 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.175544 2537 scope.go:117] "RemoveContainer" containerID="60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234" Jul 30 21:20:01.175800 ip-10-0-131-223 kubenswrapper[2537]: E0730 21:20:01.175769 2537 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234\": container with ID starting with 60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234 not found: ID does not exist" containerID="60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234" Jul 30 21:20:01.175860 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.175808 2537 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234"} err="failed to get container status \"60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234\": rpc error: code = NotFound desc = could not find container \"60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234\": container with ID starting with 60b1cd3719741112c0f920ed05becb506dccf6e7b30ac78dc7ac4e70eb798234 not found: ID does not exist" Jul 30 21:20:01.178684 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.178664 2537 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz"] Jul 30 21:20:01.181610 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:01.181589 2537 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6967f3-openshift-default-59888b5dd6-m57vz"] Jul 30 21:20:02.130111 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:02.130079 2537 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" path="/var/lib/kubelet/pods/dd58a576-d8cc-4953-9452-8f3ec692ee86/volumes" Jul 30 21:20:05.940434 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:05.940399 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:06.008536 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:06.008501 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:06.107716 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:06.107686 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:06.170485 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:06.170452 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:06.496229 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:06.496200 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:08.064706 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:08.064672 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:20:12.618745 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:12.618714 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:12.698241 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:12.698210 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:12.896689 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:12.896608 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:13.217629 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:13.217506 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:23.825623 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:23.825590 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:23.895876 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:23.895836 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:23.994946 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:23.994915 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:24.060322 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:24.060284 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:24.388624 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:24.388590 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:25.859331 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:25.859292 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-pgldp"] Jul 30 21:20:30.154812 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:30.154733 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:31.397894 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:31.397848 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:36.569489 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:36.569453 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5fr66_e2c4c773-765d-4d42-bc81-1c981a68da0f/manager/0.log" Jul 30 21:20:36.686186 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:36.686155 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zkpmd_d2d73de9-5c72-4e81-bfe4-4b68375853b7/manager/0.log" Jul 30 21:20:36.939472 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:36.939441 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jc9qc_69d994d8-e020-4b34-aa88-586ada797d49/server/0.log" Jul 30 21:20:37.059446 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:37.059413 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2m8b5_c5913a92-4b24-439d-bcbf-810625bf88d0/manager/0.log" Jul 30 21:20:37.316877 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:37.316798 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-f5dfd_85f353d8-4ba4-4d60-929e-686dcb6c3cc4/copy-manifests/0.log" Jul 30 21:20:37.327585 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:37.327559 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-f5dfd_85f353d8-4ba4-4d60-929e-686dcb6c3cc4/manager/0.log" Jul 30 21:20:39.300958 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:39.300917 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-w776m_52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e/manager/0.log" Jul 30 21:20:39.416268 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:39.416240 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ppb58_15ce5183-3608-4f26-9322-3e6677a6d21f/kuadrant-console-plugin/0.log" Jul 30 21:20:39.536814 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:39.536773 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-h5bbw_2958d992-efcc-4fd7-b0d3-5676d9cb6fac/manager/0.log" Jul 30 21:20:39.656640 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:39.656613 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-pgldp_6bc9b063-3816-4a3d-8f80-74d05551a229/limitador/0.log" Jul 30 21:20:39.776240 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:39.776207 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-f8ztj_abb37ac3-96f5-4880-b909-2605d9452ff9/manager/0.log" Jul 30 21:20:40.137431 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:40.137403 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czqq94_af910c78-fd9c-4dfc-96f6-876207dd3f33/istio-proxy/0.log" Jul 30 21:20:40.369165 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:40.369132 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c4ff7d78-ch2mn_f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9/kube-auth-proxy/0.log" Jul 30 21:20:40.604831 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:40.604737 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-q48mv_41aee649-d954-4e90-b0f4-be92c66efaea/istio-proxy/1.log" Jul 30 21:20:41.690735 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:41.690704 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftrq2m_9ca509ce-aa1a-493b-bee7-8e957ef1be28/main/0.log" Jul 30 21:20:41.808564 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:41.808531 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-96d6n_55d84486-6ada-49fc-99ed-ef126e46cd75/main/0.log" Jul 30 21:20:41.922832 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:41.922805 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-gppwf_07c748b4-fc2e-4669-8546-3ed8122e5cf9/main/0.log" Jul 30 21:20:47.810990 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:47.810947 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:47.876621 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:47.876589 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:20:50.632332 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:50.632305 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gr4lp_0e4765f1-4ab8-4d6c-8f93-0ee04514feb1/global-pull-secret-syncer/0.log" Jul 30 21:20:50.721235 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:50.721210 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" Jul 30 21:20:50.809789 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:50.809746 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-223.ec2.internal_7a2f016900707da51a5c4bb3e93fff86/haproxy/0.log" Jul 30 21:20:56.387891 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:56.387862 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-w776m_52733b6b-3ff3-4c0d-8fbe-2a7f812d2a3e/manager/0.log" Jul 30 21:20:56.419107 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:56.419077 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ppb58_15ce5183-3608-4f26-9322-3e6677a6d21f/kuadrant-console-plugin/0.log" Jul 30 21:20:56.525657 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:56.525618 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-h5bbw_2958d992-efcc-4fd7-b0d3-5676d9cb6fac/manager/0.log" Jul 30 21:20:56.550692 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:56.550664 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-pgldp_6bc9b063-3816-4a3d-8f80-74d05551a229/limitador/0.log" Jul 30 21:20:56.635163 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:56.635124 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-f8ztj_abb37ac3-96f5-4880-b909-2605d9452ff9/manager/0.log" Jul 30 21:20:58.447079 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:58.447052 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hxhwk_443e5a17-c7d3-405e-b103-fdee927b1a99/node-exporter/0.log" Jul 30 21:20:58.475759 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:58.475734 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hxhwk_443e5a17-c7d3-405e-b103-fdee927b1a99/kube-rbac-proxy/0.log" Jul 30 21:20:58.498420 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:20:58.498393 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hxhwk_443e5a17-c7d3-405e-b103-fdee927b1a99/init-textfile/0.log" Jul 30 21:21:01.140746 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.140710 2537 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp"] Jul 30 21:21:01.142865 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.142833 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" Jul 30 21:21:01.143043 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143030 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" Jul 30 21:21:01.143136 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143125 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59e3f0e-3bcb-4839-89fd-ff2605046ec0" containerName="main" Jul 30 21:21:01.143214 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143205 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59e3f0e-3bcb-4839-89fd-ff2605046ec0" containerName="main" Jul 30 21:21:01.143315 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143305 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" Jul 30 21:21:01.143391 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143382 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" Jul 30 21:21:01.143461 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143452 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" Jul 30 21:21:01.143527 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143518 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" Jul 30 21:21:01.143602 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143593 2537 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" Jul 30 21:21:01.143671 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143662 2537 state_mem.go:107] "Deleted CPUSet assignment" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" Jul 30 21:21:01.143972 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.143955 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" Jul 30 21:21:01.144089 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.144077 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd58a576-d8cc-4953-9452-8f3ec692ee86" containerName="istio-proxy" Jul 30 21:21:01.144175 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.144166 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" Jul 30 21:21:01.144251 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.144242 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="b663b0c9-0663-4bab-9cae-ba8c3d322bcd" containerName="istio-proxy" Jul 30 21:21:01.144329 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.144320 2537 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59e3f0e-3bcb-4839-89fd-ff2605046ec0" containerName="main" Jul 30 21:21:01.148095 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.148069 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.151237 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.151210 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dkmk7\"/\"openshift-service-ca.crt\"" Jul 30 21:21:01.151237 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.151213 2537 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dkmk7\"/\"default-dockercfg-xzmq2\"" Jul 30 21:21:01.151447 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.151253 2537 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dkmk7\"/\"kube-root-ca.crt\"" Jul 30 21:21:01.153115 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.153094 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp"] Jul 30 21:21:01.250013 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.249970 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762v5\" (UniqueName: \"kubernetes.io/projected/3708697b-d1c0-47af-94a2-90256e1690cd-kube-api-access-762v5\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.250013 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.250014 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-lib-modules\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.250300 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.250088 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-proc\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.250300 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.250217 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-podres\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.250300 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.250240 2537 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-sys\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351029 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.350987 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-lib-modules\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351188 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351046 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-proc\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351188 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351090 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-podres\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351188 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351107 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-sys\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351188 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351136 2537 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762v5\" (UniqueName: \"kubernetes.io/projected/3708697b-d1c0-47af-94a2-90256e1690cd-kube-api-access-762v5\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351358 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351191 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-lib-modules\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351358 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351219 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-sys\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351358 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351191 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-proc\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.351358 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.351221 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3708697b-d1c0-47af-94a2-90256e1690cd-podres\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.358860 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.358837 2537 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-762v5\" (UniqueName: \"kubernetes.io/projected/3708697b-d1c0-47af-94a2-90256e1690cd-kube-api-access-762v5\") pod \"perf-node-gather-daemonset-x75mp\" (UID: \"3708697b-d1c0-47af-94a2-90256e1690cd\") " pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.458986 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.458888 2537 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:01.590633 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:01.590607 2537 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp"] Jul 30 21:21:01.592712 ip-10-0-131-223 kubenswrapper[2537]: W0730 21:21:01.592684 2537 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3708697b_d1c0_47af_94a2_90256e1690cd.slice/crio-20c7a6a239afc094f8061e9eac48f7f2863e23af4e76855ef1a38588cdab68e5 WatchSource:0}: Error finding container 20c7a6a239afc094f8061e9eac48f7f2863e23af4e76855ef1a38588cdab68e5: Status 404 returned error can't find the container with id 20c7a6a239afc094f8061e9eac48f7f2863e23af4e76855ef1a38588cdab68e5 Jul 30 21:21:02.451226 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:02.451190 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" event={"ID":"3708697b-d1c0-47af-94a2-90256e1690cd","Type":"ContainerStarted","Data":"d459f8dcb091405adbe45d20b52850311cda25d251070a69f7f18e0efe013f27"} Jul 30 21:21:02.451685 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:02.451242 2537 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" event={"ID":"3708697b-d1c0-47af-94a2-90256e1690cd","Type":"ContainerStarted","Data":"20c7a6a239afc094f8061e9eac48f7f2863e23af4e76855ef1a38588cdab68e5"} Jul 30 21:21:02.451685 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:02.451278 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:02.468599 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:02.468543 2537 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" podStartSLOduration=1.468525632 podStartE2EDuration="1.468525632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 21:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:21:02.467674371 +0000 UTC m=+3588.727986898" watchObservedRunningTime="2026-07-30 21:21:02.468525632 +0000 UTC m=+3588.728838158" Jul 30 21:21:03.532382 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:03.532354 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sphlb_ea1f0009-1ced-4345-aad3-618ab9cf8546/dns/0.log" Jul 30 21:21:03.554685 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:03.554652 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sphlb_ea1f0009-1ced-4345-aad3-618ab9cf8546/kube-rbac-proxy/0.log" Jul 30 21:21:03.628129 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:03.628093 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2w9bg_fa158ca8-9798-4ad3-b302-a69eb51e1295/dns-node-resolver/0.log" Jul 30 21:21:04.211599 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:04.211567 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-777fbbd874-dkkms_ee5f1cc0-1374-4d34-9c52-654943b71645/registry/0.log" Jul 30 21:21:04.232068 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:04.232046 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b9lwj_16f37bef-efea-4c62-b4ca-fcb69b4ecaad/node-ca/0.log" Jul 30 21:21:05.227655 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:05.227619 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czqq94_af910c78-fd9c-4dfc-96f6-876207dd3f33/istio-proxy/0.log" Jul 30 21:21:05.418692 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:05.418664 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c4ff7d78-ch2mn_f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9/kube-auth-proxy/0.log" Jul 30 21:21:05.472161 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:05.472129 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-q48mv_41aee649-d954-4e90-b0f4-be92c66efaea/istio-proxy/0.log" Jul 30 21:21:05.478941 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:05.478887 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-q48mv_41aee649-d954-4e90-b0f4-be92c66efaea/istio-proxy/1.log" Jul 30 21:21:06.395580 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:06.395501 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s99zs_1eb47f98-77c5-46a2-a3af-edbebbfb9d82/serve-healthcheck-canary/0.log" Jul 30 21:21:06.917033 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:06.917004 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2jbj_aab8496e-e598-4ff6-b83c-c35c6aeb6320/kube-rbac-proxy/0.log" Jul 30 21:21:06.938953 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:06.938926 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2jbj_aab8496e-e598-4ff6-b83c-c35c6aeb6320/exporter/0.log" Jul 30 21:21:06.963451 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:06.963426 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2jbj_aab8496e-e598-4ff6-b83c-c35c6aeb6320/extractor/0.log" Jul 30 21:21:08.464914 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:08.464886 2537 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dkmk7/perf-node-gather-daemonset-x75mp" Jul 30 21:21:09.239947 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:09.239923 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5fr66_e2c4c773-765d-4d42-bc81-1c981a68da0f/manager/0.log" Jul 30 21:21:09.463751 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:09.463719 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zkpmd_d2d73de9-5c72-4e81-bfe4-4b68375853b7/manager/0.log" Jul 30 21:21:09.636132 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:09.636105 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jc9qc_69d994d8-e020-4b34-aa88-586ada797d49/server/0.log" Jul 30 21:21:09.716103 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:09.716080 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2m8b5_c5913a92-4b24-439d-bcbf-810625bf88d0/manager/0.log" Jul 30 21:21:09.873385 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:09.873357 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-f5dfd_85f353d8-4ba4-4d60-929e-686dcb6c3cc4/manager/0.log" Jul 30 21:21:09.897003 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:09.896941 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fb4f68b-f5dfd_85f353d8-4ba4-4d60-929e-686dcb6c3cc4/copy-manifests/0.log" Jul 30 21:21:11.092235 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:11.092153 2537 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-97pmk_0c15f8d3-eeb8-4d1e-8705-ac2ae2344cb9/konnectivity-agent/0.log" limit=1024 Jul 30 21:21:11.439756 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:11.439729 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-j5q68_d4d19509-3783-44dd-9d0a-9d4d46a63320/openshift-lws-operator/0.log" Jul 30 21:21:12.123613 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:12.123578 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czqq94_af910c78-fd9c-4dfc-96f6-876207dd3f33/istio-proxy/0.log" Jul 30 21:21:12.319703 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:12.319680 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c4ff7d78-ch2mn_f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9/kube-auth-proxy/0.log" Jul 30 21:21:12.358039 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:12.358003 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-q48mv_41aee649-d954-4e90-b0f4-be92c66efaea/istio-proxy/0.log" Jul 30 21:21:12.365022 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:12.365000 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-q48mv_41aee649-d954-4e90-b0f4-be92c66efaea/istio-proxy/1.log" Jul 30 21:21:13.869606 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:13.869573 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799czqq94_af910c78-fd9c-4dfc-96f6-876207dd3f33/istio-proxy/0.log" Jul 30 21:21:14.067269 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:14.067218 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55c4ff7d78-ch2mn_f8584ac5-cd46-4ee1-b320-fbe5e6dbb7a9/kube-auth-proxy/0.log" Jul 30 21:21:14.107625 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:14.107599 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-q48mv_41aee649-d954-4e90-b0f4-be92c66efaea/istio-proxy/0.log" Jul 30 21:21:14.114325 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:14.114301 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-q48mv_41aee649-d954-4e90-b0f4-be92c66efaea/istio-proxy/1.log" Jul 30 21:21:17.809604 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:17.809576 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k5h49_3ce791b2-1232-4fbe-9ffe-523b7263a619/kube-multus-additional-cni-plugins/0.log" Jul 30 21:21:17.831039 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:17.831016 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k5h49_3ce791b2-1232-4fbe-9ffe-523b7263a619/egress-router-binary-copy/0.log" Jul 30 21:21:17.852464 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:17.852441 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k5h49_3ce791b2-1232-4fbe-9ffe-523b7263a619/cni-plugins/0.log" Jul 30 21:21:17.875273 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:17.875245 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k5h49_3ce791b2-1232-4fbe-9ffe-523b7263a619/bond-cni-plugin/0.log" Jul 30 21:21:17.895948 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:17.895925 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k5h49_3ce791b2-1232-4fbe-9ffe-523b7263a619/routeoverride-cni/0.log" Jul 30 21:21:17.919023 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:17.919002 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k5h49_3ce791b2-1232-4fbe-9ffe-523b7263a619/whereabouts-cni-bincopy/0.log" Jul 30 21:21:17.940826 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:17.940806 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k5h49_3ce791b2-1232-4fbe-9ffe-523b7263a619/whereabouts-cni/0.log" Jul 30 21:21:18.368953 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:18.368918 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s7jf2_67bcbfeb-4460-4bc4-bc79-0cfcc04f0936/kube-multus/0.log" Jul 30 21:21:18.487275 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:18.487247 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q2k9d_0de7db11-2876-47d9-bbd3-c2a8faf8bf8d/network-metrics-daemon/0.log" Jul 30 21:21:18.507117 ip-10-0-131-223 kubenswrapper[2537]: I0730 21:21:18.507092 2537 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q2k9d_0de7db11-2876-47d9-bbd3-c2a8faf8bf8d/kube-rbac-proxy/0.log"